scheduler changes for v3.15
Subsystems (22)
Linus,
Please pull the latest sched-core-for-linus git tree from:
git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git sched-core-for-linus
# HEAD: 6037dd1a49f95092824fa8ba75c717ff7805e317 sched: Clean up the task_hot() function
Bigger changes:
* sched/idle restructuring: they are WIP preparation for deeper integration between the scheduler and idle state selection, by Nicolas Pitre.
* add NUMA scheduling pseudo-interleaving, by Rik van Riel.
* optimize cgroup context switches, by Peter Zijlstra.
* RT scheduling enhancements, by Thomas Gleixner.
The rest is smaller changes, non-urgnt fixes and cleanups.
Thanks,
Ingo
------------------>
Alex Shi (2):
sched: Add statistic for newidle load balance cost
sched: Clean up the task_hot() function
Bjorn Helgaas (2):
sched: Remove unused mc_capable() and smt_capable() sparc64, sched: Remove unused sparc64_multi_core
Daniel Lezcano (3):
sched: Remove 'cpu' parameter from idle_balance()
sched: Fix race in idle_balance()
sched: Move rq->idle_stamp up to the core
Dietmar Eggemann (2):
sched: Delete is_same_group() outside CONFIG_FAIR_GROUP_SCHED
sched: Put rq's sched_avg under CONFIG_FAIR_GROUP_SCHED
Dongsheng Yang (10):
sched: Move the priority specific bits into a new header file
sched: Expose some macros related to priority
sched: Implement task_nice() as static inline function sched/prio: Use DEFAULT_PRIO to define NICE_TO_PRIO() and PRIO_TO_NICE() sched/prio: Add 3 macros of MAX_NICE, MIN_NICE and NICE_WIDTH in prio.h
rcu: Use MAX_NICE to replace hardcoding of 19
sched: Replace hardcoding of -20 and 19 with MIN_NICE and MAX_NICE
sys: Replace hardcoding of -20 and 19 with MIN_NICE and MAX_NICE
workqueue: Replace hardcoding of -20 and 19 with MIN_NICE and MAX_NICE
trace: Replace hardcoding of 19 with MAX_NICE
Jeremy Kerr (1):
powerpc/spufs: Remove MAX_USER_PRIO define
Kirill Tkhai (5):
sched/deadline: Skip in switched_to_dl() if task is current sched/rt: Fix picking RT and DL tasks from empty queue sched/fair: Push down check for high priority class task into idle_balance() sched/core: Fix endless loop in pick_next_task()
sched/fair: Fix endless loop in idle_balance()
Li Zefan (2):
sched/rt: Remove 'leaf_rt_rq_list' from 'struct rq' sched/rt: Make init_sched_rt_calss() __init
Michael wang (1):
sched/fair: Reset se-depth when task switched to FAIR
Mike Galbraith (2):
sched, nohz: Exclude isolated cores from load balancing sched/numa: Move task_numa_free() to __put_task_struct()
Morten Rasmussen (1):
sched: Fix select_task_rq_fair() description comments
Nicolas Pitre (8):
sched/idle: Move the cpuidle entry point to the generic idle loop sched/idle, ARM: Remove redundant cpuidle_idle_call() sched/idle, PPC: Remove redundant cpuidle_idle_call() sched/idle, SH: Remove redundant cpuidle_idle_call() sched/idle, x86: Remove redundant cpuidle_idle_call() sched/idle: Move cpu/idle.c to sched/idle.c cpuidle/powernv: Remove redundant cpuidle_idle_call() cpuidle/arm64: Remove redundant cpuidle_idle_call()
Peter Zijlstra (11):
sched/fair: Track cgroup depth
sched: Push put_prev_task() into pick_next_task() sched/fair: Clean up the __clear_buddies_*() functions sched/fair: Optimize cgroup pick_next_task_fair()
sched: Clean up idle task SMP logic
sched: Push down pre_schedule() and idle_balance() sched/fair: Remove idle_balance() declaration in sched.h
sched: Fix hotplug task migration
sched: Remove some #ifdeffery
sched/idle: Remove stale old file
sched: Guarantee task priority in pick_next_task()
Preeti U Murthy (1):
cpuidle/pseries: Fix fallout caused due to cleanup in pseries cpuidle backend driver
Rik van Riel (9):
sched/numa, mm: Remove p->numa_migrate_deferred
sched/numa: Rename p->numa_faults to numa_faults_memory
sched/numa: Track from which nodes NUMA faults are triggered sched/numa: Build per numa_group active node mask from numa_faults_cpu statistics sched/numa, mm: Use active_nodes nodemask to limit numa migrations sched/numa: Normalize faults_cpu stats and weigh by CPU use sched/numa: Do statistics calculation using local variables only sched/numa: Rename variables in task_numa_fault()
sched/numa: Turn some magic numbers into #defines
Steven Rostedt (1):
sched: Fix broken setscheduler()
Thomas Gleixner (6):
sched: Init idle->on_rq in init_idle()
sched: Check for idle task in might_sleep()
sched: Add better debug output for might_sleep()
sched: Adjust p->sched_reset_on_fork when nothing else changes
sched: Queue RT tasks to head when prio drops
sched: Consider pi boosting in setscheduler()
Vincent Guittot (1):
sched: Remove double calculation in fix_small_imbalance()
Documentation/sysctl/kernel.txt | 10 +-
arch/arm/include/asm/topology.h | 3 -
arch/arm/kernel/process.c | 16 +-
arch/arm64/kernel/process.c | 7 +-
arch/ia64/include/asm/topology.h | 1 -
arch/mips/include/asm/topology.h | 4 -
arch/powerpc/include/asm/topology.h | 1 -
arch/powerpc/platforms/cell/spufs/sched.c | 1 -
arch/powerpc/platforms/powernv/setup.c | 13 +-
arch/powerpc/platforms/pseries/setup.c | 34 +-
arch/sh/kernel/idle.c | 4 +-
arch/sparc/include/asm/smp_64.h | 1 -
arch/sparc/include/asm/topology_64.h | 2 -
arch/sparc/kernel/mdesc.c | 4 -
arch/sparc/kernel/prom_64.c | 3 -
arch/sparc/kernel/smp_64.c | 2 -
arch/x86/include/asm/topology.h | 6 -
arch/x86/kernel/process.c | 5 +-
drivers/cpuidle/cpuidle-powernv.c | 5 +
drivers/cpuidle/cpuidle-pseries.c | 6 +
include/linux/sched.h | 44 ++-
include/linux/sched/prio.h | 44 +++
include/linux/sched/rt.h | 26 +-
kernel/Makefile | 1 -
kernel/cpu/Makefile | 1 -
kernel/fork.c | 1 +
kernel/locking/rtmutex.c | 12 +
kernel/rcu/torture.c | 8 +-
kernel/sched/Makefile | 2 +-
kernel/sched/auto_group.c | 2 +-
kernel/sched/core.c | 207 +++++++----
kernel/sched/cputime.c | 4 +-
kernel/sched/deadline.c | 56 ++-
kernel/sched/debug.c | 7 +-
kernel/sched/fair.c | 600 +++++++++++++++++++++++-------
kernel/{cpu => sched}/idle.c | 7 +-
kernel/sched/idle_task.c | 25 +-
kernel/sched/rt.c | 102 +++--
kernel/sched/sched.h | 65 ++--
kernel/sched/stop_task.c | 15 +-
kernel/sys.c | 8 +-
kernel/sysctl.c | 7 -
kernel/trace/ring_buffer_benchmark.c | 6 +-
kernel/workqueue.c | 2 +-
mm/mempolicy.c | 74 +---
45 files changed, 913 insertions(+), 541 deletions(-)
create mode 100644 include/linux/sched/prio.h
delete mode 100644 kernel/cpu/Makefile
rename kernel/{cpu => sched}/idle.c (95%)Documentation/sysctl/kernel.txt +1 -9
@@ -441,8 +441,7 @@ feature should be disabled. Otherwise, if the system overhead from the feature is too high then the rate the kernel samples for NUMA hinting faults may be controlled by the numa_balancing_scan_period_min_ms, numa_balancing_scan_delay_ms, numa_balancing_scan_period_max_ms, -numa_balancing_scan_size_mb, numa_balancing_settle_count sysctls and -numa_balancing_migrate_deferred. +numa_balancing_scan_size_mb, and numa_balancing_settle_count sysctls. ==============================================================
@@ -483,13 +482,6 @@ rate for each task. numa_balancing_scan_size_mb is how many megabytes worth of pages are scanned for a given scan. -numa_balancing_migrate_deferred is how many page migrations get skipped -unconditionally, after a page migration is skipped because a page is shared -with other tasks. This reduces page migration overhead, and determines -how much stronger the "move task near its memory" policy scheduler becomes, -versus the "move memory near its task" memory management policy, for workloads -with shared memory. - ============================================================== osrelease, ostype & version:
arch/arm/include/asm/topology.h +0 -3
@@ -20,9 +20,6 @@ extern struct cputopo_arm cpu_topology[NR_CPUS];#define topology_core_cpumask (cpu) (&cpu_topology[cpu].core_sibling)#define topology_thread_cpumask (cpu) (&cpu_topology[cpu].thread_sibling) -#define mc_capable () (cpu_topology[0].socket_id != -1) -#define smt_capable () (cpu_topology[0].thread_id != -1) -void init_cpu_topology (void );void store_cpu_topology (unsigned int cpuid);const struct cpumask *cpu_coregroup_mask (int cpu);
arch/arm/kernel/process.c +5 -11
@@ -30,7 +30,6 @@#include <linux/uaccess.h> #include <linux/random.h> #include <linux/hw_breakpoint.h> -#include <linux/cpuidle.h> #include <linux/leds.h> #include <linux/reboot.h>
@@ -133,7 +132,11 @@ EXPORT_SYMBOL_GPL(arm_pm_restart);void (*arm_pm_idle)(void ); -static void default_idle (void ) +/* +* Called from the core idle loop. +*/ + +void arch_cpu_idle (void ) {if (arm_pm_idle)arm_pm_idle ();
@@ -168,15 +171,6 @@ void arch_cpu_idle_dead(void)#endif /* -* Called from the core idle loop. -*/ -void arch_cpu_idle (void ) -{ -if (cpuidle_idle_call ()) -default_idle (); -} - -/* * Called by kexec, immediately prior to machine_kexec(). * * This must completely disable all secondary CPUs; simply causing those CPUs
arch/arm64/kernel/process.c +2 -5
@@ -33,7 +33,6 @@#include <linux/kallsyms.h> #include <linux/init.h> #include <linux/cpu.h> -#include <linux/cpuidle.h> #include <linux/elfcore.h> #include <linux/pm.h> #include <linux/tick.h>
@@ -94,10 +93,8 @@ void arch_cpu_idle(void)* This should do all the clock switching and wait for interrupt * tricks */ -if (cpuidle_idle_call ()) { -cpu_do_idle (); -local_irq_enable (); - } +cpu_do_idle (); +local_irq_enable (); }#ifdef CONFIG_HOTPLUG_CPU
arch/ia64/include/asm/topology.h +0 -1
@@ -77,7 +77,6 @@ void build_cpu_to_node_map(void);#define topology_core_id (cpu) (cpu_data(cpu)->core_id)#define topology_core_cpumask (cpu) (&cpu_core_map[cpu])#define topology_thread_cpumask (cpu) (&per_cpu(cpu_sibling_map, cpu)) -#define smt_capable () (smp_num_siblings > 1)#endif extern void arch_fix_phys_package_id (int num,u32 slot);
arch/mips/include/asm/topology.h +0 -4
@@ -10,8 +10,4 @@#include <topology.h> -#ifdef CONFIG_SMP -#define smt_capable () (smp_num_siblings > 1) -#endif -#endif /* __ASM_TOPOLOGY_H */
arch/powerpc/include/asm/topology.h +0 -1
@@ -99,7 +99,6 @@ static inline int prrn_is_enabled(void)#ifdef CONFIG_SMP #include <asm/cputable.h> -#define smt_capable () (cpu_has_feature(CPU_FTR_SMT))#ifdef CONFIG_PPC64 #include <asm/smp.h>
arch/powerpc/platforms/cell/spufs/sched.c +0 -1
@@ -83,7 +83,6 @@ static struct timer_list spuloadavg_timer;#define MIN_SPU_TIMESLICE max(5 * HZ / (1000 * SPUSCHED_TICK), 1)#define DEF_SPU_TIMESLICE (100 * HZ / (1000 * SPUSCHED_TICK)) -#define MAX_USER_PRIO (MAX_PRIO - MAX_RT_PRIO)#define SCALE_PRIO (x, prio) \ max(x * (MAX_PRIO - prio) / (MAX_USER_PRIO / 2), MIN_SPU_TIMESLICE)
arch/powerpc/platforms/powernv/setup.c +1 -12
@@ -26,7 +26,6 @@#include <linux/of_fdt.h> #include <linux/interrupt.h> #include <linux/bug.h> -#include <linux/cpuidle.h> #include <linux/pci.h> #include <asm/machdep.h>
@@ -225,16 +224,6 @@ static int __init pnv_probe(void)return 1 ; } -void powernv_idle (void ) -{ -/* Hook to cpuidle framework if available, else -* call on default platform idle code -*/ -if (cpuidle_idle_call ()) { -power7_idle (); - } -} -define_machine (powernv) { .name = "PowerNV", .probe = pnv_probe,
@@ -244,7 +233,7 @@ define_machine(powernv) { .show_cpuinfo = pnv_show_cpuinfo, .progress = pnv_progress, .machine_shutdown = pnv_shutdown, - .power_save = powernv_idle, + .power_save = power7_idle, .calibrate_decr = generic_calibrate_decr, .dma_set_mask = pnv_dma_set_mask,#ifdef CONFIG_KEXEC
arch/powerpc/platforms/pseries/setup.c +14 -20
@@ -39,7 +39,6 @@#include <linux/irq.h> #include <linux/seq_file.h> #include <linux/root_dev.h> -#include <linux/cpuidle.h> #include <linux/of.h> #include <linux/kexec.h>
@@ -356,29 +355,24 @@ early_initcall(alloc_dispatch_log_kmem_cache);static void pseries_lpar_idle (void ) { -/* This would call on the cpuidle framework, and the back-end pseries -* driver to go to idle states +/* +* Default handler to go into low thread priority and possibly +* low power mode by cedeing processor to hypervisor */ -if (cpuidle_idle_call ()) { -/* On error, execute default handler -* to go into low thread priority and possibly -* low power mode by cedeing processor to hypervisor -*/ -/* Indicate to hypervisor that we are idle. */ -get_lppaca ()->idle =1 ; +/* Indicate to hypervisor that we are idle. */ +get_lppaca ()->idle =1 ; -/* -* Yield the processor to the hypervisor. We return if -* an external interrupt occurs (which are driven prior -* to returning here) or if a prod occurs from another -* processor. When returning here, external interrupts -* are enabled. -*/ -cede_processor (); +/* +* Yield the processor to the hypervisor. We return if +* an external interrupt occurs (which are driven prior +* to returning here) or if a prod occurs from another +* processor. When returning here, external interrupts +* are enabled. +*/ +cede_processor (); -get_lppaca ()->idle =0 ; - } +get_lppaca ()->idle =0 ; }/*
arch/sh/kernel/idle.c +1 -3
@@ -16,7 +16,6 @@#include <linux/thread_info.h> #include <linux/irqflags.h> #include <linux/smp.h> -#include <linux/cpuidle.h> #include <linux/atomic.h> #include <asm/pgalloc.h> #include <asm/smp.h>
@@ -40,8 +39,7 @@ void arch_cpu_idle_dead(void)void arch_cpu_idle (void ) { -if (cpuidle_idle_call ()) -sh_idle (); +sh_idle (); }void __init select_idle_routine (void )
arch/sparc/include/asm/smp_64.h +0 -1
@@ -32,7 +32,6 @@DECLARE_PER_CPU (cpumask_t, cpu_sibling_map);extern cpumask_t cpu_core_map[NR_CPUS ]; -extern int sparc64_multi_core;extern void arch_send_call_function_single_ipi (int cpu);extern void arch_send_call_function_ipi_mask (const struct cpumask *mask);
arch/sparc/include/asm/topology_64.h +0 -2
@@ -42,8 +42,6 @@ static inline int pcibus_to_node(struct pci_bus *pbus)#define topology_core_id (cpu) (cpu_data(cpu).core_id)#define topology_core_cpumask (cpu) (&cpu_core_map[cpu])#define topology_thread_cpumask (cpu) (&per_cpu(cpu_sibling_map, cpu)) -#define mc_capable () (sparc64_multi_core) -#define smt_capable () (sparc64_multi_core)#endif /* CONFIG_SMP */ extern cpumask_t cpu_core_map[NR_CPUS ];
arch/sparc/kernel/mdesc.c +0 -4
@@ -896,10 +896,6 @@ void mdesc_fill_in_cpu_data(cpumask_t *mask)mdesc_iterate_over_cpus (fill_in_one_cpu,NULL , mask); -#ifdef CONFIG_SMP - sparc64_multi_core =1 ; -#endif - hp =mdesc_grab ();set_core_ids (hp);
arch/sparc/kernel/prom_64.c +0 -3
@@ -555,9 +555,6 @@ static void *fill_in_one_cpu(struct device_node *dp, int cpuid, int arg)cpu_data (cpuid).core_id = portid +1 ;cpu_data (cpuid).proc_id = portid; -#ifdef CONFIG_SMP - sparc64_multi_core =1 ; -#endif }else {cpu_data (cpuid).dcache_size =of_getintprop_default (dp,"dcache-size" ,16 *1024 );
arch/sparc/kernel/smp_64.c +0 -2
@@ -53,8 +53,6 @@#include "cpumap.h" -int sparc64_multi_core__read_mostly ; -DEFINE_PER_CPU (cpumask_t, cpu_sibling_map) =CPU_MASK_NONE ;cpumask_t cpu_core_map[NR_CPUS ]__read_mostly = { [0 ...NR_CPUS -1 ] =CPU_MASK_NONE };
arch/x86/include/asm/topology.h +0 -6
@@ -133,12 +133,6 @@ static inline void arch_fix_phys_package_id(int num, u32 slot)struct pci_bus ;void x86_pci_root_bus_resources (int bus,struct list_head *resources); -#ifdef CONFIG_SMP -#define mc_capable () ((boot_cpu_data.x86_max_cores > 1) && \ - (cpumask_weight(cpu_core_mask(0)) != nr_cpu_ids)) -#define smt_capable () (smp_num_siblings > 1) -#endif -#ifdef CONFIG_NUMA extern int get_mp_bus_to_node (int busnum);extern void set_mp_bus_to_node (int busnum,int node);
arch/x86/kernel/process.c +1 -4
@@ -298,10 +298,7 @@ void arch_cpu_idle_dead(void)*/ void arch_cpu_idle (void ) { -if (cpuidle_idle_call ()) -x86_idle (); -else -local_irq_enable (); +x86_idle (); }/*
drivers/cpuidle/cpuidle-powernv.c +5 -0
@@ -14,6 +14,7 @@#include <asm/machdep.h> #include <asm/firmware.h> +#include <asm/runlatch.h> struct cpuidle_driver powernv_idle_driver = { .name ="powernv_idle" ,
@@ -30,12 +31,14 @@ static int snooze_loop(struct cpuidle_device *dev,local_irq_enable ();set_thread_flag (TIF_POLLING_NRFLAG ); +ppc64_runlatch_off ();while (!need_resched ()) {HMT_low ();HMT_very_low (); }HMT_medium (); +ppc64_runlatch_on ();clear_thread_flag (TIF_POLLING_NRFLAG );smp_mb ();return index;
@@ -45,7 +48,9 @@ static int nap_loop(struct cpuidle_device *dev,struct cpuidle_driver *drv, int index) { +ppc64_runlatch_off ();power7_idle (); +ppc64_runlatch_on ();return index; }
drivers/cpuidle/cpuidle-pseries.c +6 -0
@@ -17,6 +17,7 @@#include <asm/reg.h> #include <asm/machdep.h> #include <asm/firmware.h> +#include <asm/runlatch.h> #include <asm/plpar_wrappers.h> struct cpuidle_driver pseries_idle_driver = {
@@ -29,6 +30,7 @@ static struct cpuidle_state *cpuidle_state_table;static inline void idle_loop_prolog (unsigned long *in_purr) { +ppc64_runlatch_off (); *in_purr =mfspr (SPRN_PURR );/* * Indicate to the HV that we are idle. Now would be
@@ -45,6 +47,10 @@ static inline void idle_loop_epilog(unsigned long in_purr) wait_cycles +=mfspr (SPRN_PURR ) - in_purr;get_lppaca ()->wait_state_cycles =cpu_to_be64 (wait_cycles);get_lppaca ()->idle =0 ; + +if (irqs_disabled ()) +local_irq_enable (); +ppc64_runlatch_on (); } static int snooze_loop(struct cpuidle_device *dev,
include/linux/sched.h +36 -8
99 lines not on this page · raw message
include/linux/sched/prio.h new file +44 -0
44 lines not on this page · raw message
include/linux/sched/rt.h +8 -18
@@ -1,24 +1,7 @@#ifndef _SCHED_RT_H#define _SCHED_RT_H -/* -* Priority of a process goes from 0..MAX_PRIO-1, valid RT -* priority is 0..MAX_RT_PRIO-1, and SCHED_NORMAL/SCHED_BATCH -* tasks are in the range MAX_RT_PRIO..MAX_PRIO-1. Priority -* values are inverted: lower p->prio value means higher priority. -* -* The MAX_USER_RT_PRIO value allows the actual maximum -* RT priority to be separate from the value exported to -* user-space. This allows kernel threads to set their -* priority to a value higher than any user task. Note: -* MAX_RT_PRIO must not be smaller than MAX_USER_RT_PRIO. -*/ - -#define MAX_USER_RT_PRIO 100 -#define MAX_RT_PRIO MAX_USER_RT_PRIO - -#define MAX_PRIO (MAX_RT_PRIO + 40) -#define DEFAULT_PRIO (MAX_RT_PRIO + 20) +#include <linux/sched/prio.h> static inline int rt_prio (int prio) {
@@ -35,6 +18,7 @@ static inline int rt_task(struct task_struct *p)#ifdef CONFIG_RT_MUTEXES extern int rt_mutex_getprio (struct task_struct *p);extern void rt_mutex_setprio (struct task_struct *p,int prio); +extern int rt_mutex_check_prio (struct task_struct *task,int newprio);extern struct task_struct *rt_mutex_get_top_task (struct task_struct *task);extern void rt_mutex_adjust_pi (struct task_struct *p);static inline bool tsk_is_pi_blocked (struct task_struct *tsk)
@@ -46,6 +30,12 @@ static inline int rt_mutex_getprio(struct task_struct *p) {return p->normal_prio; } + +static inline int rt_mutex_check_prio (struct task_struct *task,int newprio) +{ +return 0 ; +} +static inline struct task_struct *rt_mutex_get_top_task (struct task_struct *task) {return NULL ;
kernel/Makefile +0 -1
@@ -22,7 +22,6 @@ obj-y += sched/ obj-y += locking/ obj-y += power/ obj-y += printk/ -obj-y += cpu/ obj-y += irq/ obj-y += rcu/
kernel/cpu/Makefile deleted +0 -1
@@ -1 +0,0 @@ -obj-y = idle.o
kernel/fork.c +1 -0
@@ -237,6 +237,7 @@ void __put_task_struct(struct task_struct *tsk)WARN_ON (atomic_read (&tsk->usage));WARN_ON (tsk == current); +task_numa_free (tsk);security_task_free (tsk);exit_creds (tsk);delayacct_tsk_free (tsk);
kernel/locking/rtmutex.c +12 -0
@@ -213,6 +213,18 @@ struct task_struct *rt_mutex_get_top_task(struct task_struct *task) }/* +* Called by sched_setscheduler() to check whether the priority change +* is overruled by a possible priority boosting. +*/ +int rt_mutex_check_prio (struct task_struct *task,int newprio) +{ +if (!task_has_pi_waiters (task)) +return 0 ; + +return task_top_pi_waiter (task)->task->prio <= newprio; +} + +/* * Adjust the priority of a task, after its pi_waiters got modified. * * This can be both boosting and unboosting. task->pi_lock must be held.
kernel/rcu/torture.c +4 -4
@@ -805,7 +805,7 @@ rcu_torture_writer(void *arg)static DEFINE_RCU_RANDOM (rand);VERBOSE_PRINTK_STRING ("rcu_torture_writer task started" ); -set_user_nice (current,19 ); +set_user_nice (current,MAX_NICE ); do { schedule_timeout_uninterruptible(1);
@@ -871,7 +871,7 @@ rcu_torture_fakewriter(void *arg)DEFINE_RCU_RANDOM (rand);VERBOSE_PRINTK_STRING ("rcu_torture_fakewriter task started" ); -set_user_nice (current,19 ); +set_user_nice (current,MAX_NICE ); do { schedule_timeout_uninterruptible(1 + rcu_random(&rand)%10);
@@ -987,7 +987,7 @@ rcu_torture_reader(void *arg)unsigned long long ts;VERBOSE_PRINTK_STRING ("rcu_torture_reader task started" ); -set_user_nice (current,19 ); +set_user_nice (current,MAX_NICE );if (irqreader && cur_ops->irq_capable)setup_timer_on_stack (&t, rcu_torture_timer,0 );
@@ -1584,7 +1584,7 @@ static int rcu_torture_barrier_cbs(void *arg)init_rcu_head_on_stack (&rcu);VERBOSE_PRINTK_STRING ("rcu_torture_barrier_cbs task started" ); -set_user_nice (current,19 ); +set_user_nice (current,MAX_NICE ); do { wait_event(barrier_cbs_wq[myid], (newphase =
kernel/sched/Makefile +1 -1
@@ -13,7 +13,7 @@ endif obj-y += core.o proc.o clock.o cputime.o obj-y += idle_task.o fair.o rt.o deadline.o stop_task.o -obj-y += wait.o completion.o +obj-y += wait.o completion.o idle.o obj-$(CONFIG_SMP) += cpupri.o cpudeadline.o obj-$(CONFIG_SCHED_AUTOGROUP) += auto_group.o obj-$(CONFIG_SCHEDSTATS) += stats.o
kernel/sched/auto_group.c +1 -1
@@ -203,7 +203,7 @@ int proc_sched_autogroup_set_nice(struct task_struct *p, int nice)struct autogroup *ag;int err; -if (nice <-20 || nice >19 ) +if (nice <MIN_NICE || nice >MAX_NICE )return -EINVAL ; err =security_task_setnice (current, nice);
kernel/sched/core.c +126 -81
431 lines not on this page · raw message
kernel/sched/cputime.c +2 -2
@@ -142,7 +142,7 @@ void account_user_time(struct task_struct *p, cputime_t cputime, p->utimescaled += cputime_scaled;account_group_user_time (p, cputime); - index = (TASK_NICE (p) >0 ) ?CPUTIME_NICE :CPUTIME_USER ; + index = (task_nice (p) >0 ) ?CPUTIME_NICE :CPUTIME_USER ;/* Add user time to cpustat. */ task_group_account_field (p, index, (__force u64 ) cputime);
@@ -169,7 +169,7 @@ static void account_guest_time(struct task_struct *p, cputime_t cputime, p->gtime += cputime;/* Add guest time to cpustat. */ -if (TASK_NICE (p) >0 ) { +if (task_nice (p) >0 ) { cpustat[CPUTIME_NICE ] += (__force u64 ) cputime; cpustat[CPUTIME_GUEST_NICE ] += (__force u64 ) cputime; }else {
kernel/sched/deadline.c +41 -15
119 lines not on this page · raw message
kernel/sched/debug.c +4 -3
@@ -321,6 +321,7 @@ do { \P (sched_goidle);#ifdef CONFIG_SMP P64 (avg_idle); +P64 (max_idle_balance_cost);#endif P (ttwu_count);
@@ -533,15 +534,15 @@ static void sched_show_numa(struct task_struct *p, struct seq_file *m)unsigned long nr_faults =-1 ;int cpu_current, home_node; -if (p->numa_faults) - nr_faults = p->numa_faults[2 *node + i]; +if (p->numa_faults_memory) + nr_faults = p->numa_faults_memory[2 *node + i]; cpu_current = !i ? (task_node (p) == node) : (pol &&node_isset (node, pol->v.nodes)); home_node = (p->numa_preferred_nid == node); -SEQ_printf (m,"numa_faults, %d, %d, %d, %d, %ld\n" , +SEQ_printf (m,"numa_faults_memory, %d, %d, %d, %d, %ld\n" , i, node, cpu_current, home_node, nr_faults); } }
kernel/sched/fair.c +468 -132
1,032 lines not on this page · raw message
kernel/sched/idle.c renamed from kernel/cpu/idle.c +5 -2
@@ -3,6 +3,7 @@*/ #include <linux/sched.h> #include <linux/cpu.h> +#include <linux/cpuidle.h> #include <linux/tick.h> #include <linux/mm.h> #include <linux/stackprotector.h>
@@ -95,8 +96,10 @@ static void cpu_idle_loop(void)if (!current_clr_polling_and_test ()) {stop_critical_timings ();rcu_idle_enter (); -arch_cpu_idle (); -WARN_ON_ONCE (irqs_disabled ()); +if (cpuidle_idle_call ()) +arch_cpu_idle (); +if (WARN_ON_ONCE (irqs_disabled ())) +local_irq_enable ();rcu_idle_exit ();start_critical_timings (); }else {
kernel/sched/idle_task.c +7 -18
@@ -13,18 +13,8 @@ select_task_rq_idle(struct task_struct *p, int cpu, int sd_flag, int flags) {return task_cpu (p);/* IDLE tasks as never migrated */ } - -static void pre_schedule_idle (struct rq *rq,struct task_struct *prev) -{ -idle_exit_fair (rq); -rq_last_tick_reset (rq); -} - -static void post_schedule_idle (struct rq *rq) -{ -idle_enter_fair (rq); -}#endif /* CONFIG_SMP */ +/* * Idle tasks are unconditionally rescheduled: */
@@ -33,13 +23,12 @@ static void check_preempt_curr_idle(struct rq *rq, struct task_struct *p, int flresched_task (rq->idle); } -static struct task_struct *pick_next_task_idle (struct rq *rq) +static struct task_struct * +pick_next_task_idle (struct rq *rq,struct task_struct *prev) { +put_prev_task (rq, prev); +schedstat_inc (rq, sched_goidle); -#ifdef CONFIG_SMP -/* Trigger the post schedule to do an idle_enter for CFS */ - rq->post_schedule =1 ; -#endif return rq->idle; }
@@ -58,6 +47,8 @@ dequeue_task_idle(struct rq *rq, struct task_struct *p, int flags)static void put_prev_task_idle (struct rq *rq,struct task_struct *prev) { +idle_exit_fair (rq); +rq_last_tick_reset (rq); }static void task_tick_idle (struct rq *rq,struct task_struct *curr,int queued)
@@ -101,8 +92,6 @@ const struct sched_class idle_sched_class = {#ifdef CONFIG_SMP .select_task_rq = select_task_rq_idle, - .pre_schedule = pre_schedule_idle, - .post_schedule = post_schedule_idle, #endif .set_curr_task = set_curr_task_idle,
kernel/sched/rt.c +65 -37
168 lines not on this page · raw message
kernel/sched/sched.h +34 -31
125 lines not on this page · raw message
kernel/sched/stop_task.c +9 -6
@@ -23,16 +23,19 @@ check_preempt_curr_stop(struct rq *rq, struct task_struct *p, int flags)/* we're never preempted */ } -static struct task_struct *pick_next_task_stop (struct rq *rq) +static struct task_struct * +pick_next_task_stop (struct rq *rq,struct task_struct *prev) {struct task_struct *stop = rq->stop; -if (stop && stop->on_rq) { - stop->se.exec_start =rq_clock_task (rq); -return stop; - } +if (!stop || !stop->on_rq) +return NULL ; -return NULL ; +put_prev_task (rq, prev); + + stop->se.exec_start =rq_clock_task (rq); + +return stop; } static void
kernel/sys.c +4 -4
@@ -174,10 +174,10 @@ SYSCALL_DEFINE3(setpriority, int, which, int, who, int, niceval)/* normalize: avoid signed division (rounding problems) */ error = -ESRCH ; -if (niceval <-20 ) - niceval =-20 ; -if (niceval >19 ) - niceval =19 ; +if (niceval <MIN_NICE ) + niceval =MIN_NICE ; +if (niceval >MAX_NICE ) + niceval =MAX_NICE ;rcu_read_lock ();read_lock (&tasklist_lock);
kernel/sysctl.c +0 -7
@@ -386,13 +386,6 @@ static struct ctl_table kern_table[] = { .proc_handler = proc_dointvec, }, { - .procname = "numa_balancing_migrate_deferred", - .data = &sysctl_numa_balancing_migrate_deferred, - .maxlen = sizeof(unsigned int), - .mode = 0644, - .proc_handler = proc_dointvec, - }, - { .procname = "numa_balancing", .data = NULL, /* filled in by handler */ .maxlen = sizeof(unsigned int),
kernel/trace/ring_buffer_benchmark.c +3 -3
@@ -40,8 +40,8 @@ static int write_iteration = 50;module_param (write_iteration, uint,0644 );MODULE_PARM_DESC (write_iteration,"# of writes between timestamp readings" ); -static int producer_nice =19 ; -static int consumer_nice =19 ; +static int producer_nice =MAX_NICE ; +static int consumer_nice =MAX_NICE ;static int producer_fifo =-1 ;static int consumer_fifo =-1 ;
@@ -308,7 +308,7 @@ static void ring_buffer_producer(void)/* Let the user know that the test is running at low priority */ if (producer_fifo <0 && consumer_fifo <0 && - producer_nice ==19 && consumer_nice ==19 ) + producer_nice ==MAX_NICE && consumer_nice ==MAX_NICE )trace_printk ("WARNING!!! This test is running at lowest priority.\n" );trace_printk ("Time: %lld (usecs)\n" , time);
kernel/workqueue.c +1 -1
@@ -3225,7 +3225,7 @@ static ssize_t wq_nice_store(struct device *dev, struct device_attribute *attr,return -ENOMEM ;if (sscanf (buf,"%d" , &attrs->nice) ==1 && - attrs->nice >=-20 && attrs->nice <=19 ) + attrs->nice >=MIN_NICE && attrs->nice <=MAX_NICE ) ret =apply_workqueue_attrs (wq, attrs);else ret = -EINVAL ;
mm/mempolicy.c +1 -73
88 lines not on this page · raw message