scheduler changes for v3.5
Subsystems (11)
Linus,
Please pull the latest sched-core-for-linus git tree from:
git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git sched-core-for-linus
HEAD: 1c2927f18576d65631d8e0ddd19e1d023183222e sched: Taint kernel with TAINT_WARN after sleep-in-atomic bug
The biggest change is the cleanup/simplification of the
load-balancer: instead of the current practice of architectures twiddling scheduler internal data structures and providing the scheduler domains in colorfully inconsistent ways, we now have generic scheduler code in kernel/sched/core.c:sched_init_numa() that looks at the architecture's node_distance() parameters and (while not fully trusting it) deducts a NUMA topology from it.
This inevitably changes balancing behavior - hopefully for the better.
There are various smaller optimizations, cleanups and fixlets as well.
Thanks,
Ingo
------------------>
Hiroshi Shimamoto (1):
sched: Update documentation and comments
Ingo Molnar (1):
x86/numa: Check for nonsensical topologies on real hw as well
Kirill Tkhai (1):
sched_rt: Avoid unnecessary dequeue and enqueue of pushable tasks in set_cpus_allowed_rt()
Konstantin Khlebnikov (1):
sched: Taint kernel with TAINT_WARN after sleep-in-atomic bug
Peter Zijlstra (15):
x86/numa: Allow specifying node_distance() for numa=fake x86/numa: Hard partition cpu topology masks on node boundaries
sched: Change rq->nr_running to unsigned int
sched/fair: Let minimally loaded cpu balance the group sched/fair: Add some serialization to the sched_domain load-balance walk sched/fair: Propagate 'struct lb_env' usage into find_busiest_group sched/numa: Rewrite the CONFIG_NUMA sched domain support sched/numa: Fix the new NUMA topology bits
sched/x86: Rewrite set_cpu_sibling_map()
sched/fair: Revert sched-domain iteration breakage
sched/numa: Don't scale the imbalance
sched/nohz: Fix rq->cpu_load[] calculations
sched/fair: Improve the ->group_imb logic
sched/debug: Fix printing large integers on 32-bit platforms
sched: Remove stale power aware scheduling remnants and dysfunctional knobs
Srivatsa S. Bhat (1):
x86/sched: Make mwait_usable() heed to "idle=" kernel parameters properly
Documentation/ABI/testing/sysfs-devices-system-cpu | 25 -- Documentation/scheduler/sched-design-CFS.txt | 6 +- Documentation/scheduler/sched-domains.txt | 4 - arch/ia64/include/asm/topology.h | 25 -- arch/mips/include/asm/mach-ip27/topology.h | 17 - arch/powerpc/include/asm/topology.h | 36 -- arch/sh/include/asm/topology.h | 25 -- arch/sparc/include/asm/topology_64.h | 19 - arch/tile/include/asm/topology.h | 26 -- arch/x86/include/asm/topology.h | 38 -- arch/x86/kernel/process.c | 8 + arch/x86/kernel/smpboot.c | 108 +++-- arch/x86/mm/numa_emulation.c | 8 +- drivers/base/cpu.c | 4 - include/linux/cpu.h | 2 - include/linux/sched.h | 49 +-- include/linux/topology.h | 42 -- kernel/sched/core.c | 420 ++++++++++--------- kernel/sched/debug.c | 12 +- kernel/sched/fair.c | 462 +++++---------------- kernel/sched/idle_task.c | 2 +- kernel/sched/rt.c | 56 ++- kernel/sched/sched.h | 8 +- tools/power/cpupower/man/cpupower-set.1 | 9 - tools/power/cpupower/utils/helpers/sysfs.c | 35 +- 25 files changed, 441 insertions(+), 1005 deletions(-)
Documentation/ABI/testing/sysfs-devices-system-cpu +0 -25
@@ -9,31 +9,6 @@ Description: /sys/devices/system/cpu/cpu#/ -What: /sys/devices/system/cpu/sched_mc_power_savings - /sys/devices/system/cpu/sched_smt_power_savings -Date: June 2006 -Contact: Linux kernel mailing list <[email protected]> -Description: Discover and adjust the kernel's multi-core scheduler support. - - Possible values are: - - 0 - No power saving load balance (default value) - 1 - Fill one thread/core/package first for long running threads - 2 - Also bias task wakeups to semi-idle cpu package for power - savings - - sched_mc_power_savings is dependent upon SCHED_MC, which is - itself architecture dependent. - - sched_smt_power_savings is dependent upon SCHED_SMT, which - is itself architecture dependent. - - The two files are independent of each other. It is possible - that one file may be present without the other. - - Introduced by git commit 5c45bf27. - - What: /sys/devices/system/cpu/kernel_max /sys/devices/system/cpu/offline /sys/devices/system/cpu/online
Documentation/scheduler/sched-design-CFS.txt +3 -3
@@ -130,7 +130,7 @@ CFS implements three scheduling policies: idle timer scheduler in order to avoid to get into priority inversion problems which would deadlock the machine. -SCHED_FIFO/_RR are implemented in sched_rt.c and are as specified by +SCHED_FIFO/_RR are implemented in sched/rt.c and are as specified by POSIX. The command chrt from util-linux-ng 2.13.1.1 can set all of these except
@@ -145,9 +145,9 @@ Classes," an extensible hierarchy of scheduler modules. These modules encapsulate scheduling policy details and are handled by the scheduler core without the core code assuming too much about them. -sched_fair.c implements the CFS scheduler described above. +sched/fair.c implements the CFS scheduler described above. -sched_rt.c implements SCHED_FIFO and SCHED_RR semantics, in a simpler way than +sched/rt.c implements SCHED_FIFO and SCHED_RR semantics, in a simpler way than the previous vanilla scheduler did. It uses 100 runqueues (for all 100 RT priority levels, instead of 140 in the previous scheduler) and it needs no expired array.
Documentation/scheduler/sched-domains.txt +0 -4
@@ -61,10 +61,6 @@ The implementor should read comments in include/linux/sched.h: struct sched_domain fields, SD_FLAG_*, SD_*_INIT to get an idea of the specifics and what to tune. -For SMT, the architecture must define CONFIG_SCHED_SMT and provide a -cpumask_t cpu_sibling_map[NR_CPUS], where cpu_sibling_map[i] is the mask of -all "i"'s siblings as well as "i" itself. - Architectures may retain the regular override the default SD_*_INIT flags while using the generic domain builder in kernel/sched.c if they wish to retain the traditional SMT->SMP->NUMA topology (or some subset of that). This
arch/ia64/include/asm/topology.h +0 -25
@@ -70,31 +70,6 @@ void build_cpu_to_node_map(void); .nr_balance_failed = 0, \ } -/* sched_domains SD_NODE_INIT for IA64 NUMA machines */ -#define SD_NODE_INIT (struct sched_domain) { \ - .parent = NULL, \ - .child = NULL, \ - .groups = NULL, \ - .min_interval = 8, \ - .max_interval = 8*(min(num_online_cpus(), 32U)), \ - .busy_factor = 64, \ - .imbalance_pct = 125, \ - .cache_nice_tries = 2, \ - .busy_idx = 3, \ - .idle_idx = 2, \ - .newidle_idx = 0, \ - .wake_idx = 0, \ - .forkexec_idx = 0, \ - .flags = SD_LOAD_BALANCE \ - | SD_BALANCE_NEWIDLE \ - | SD_BALANCE_EXEC \ - | SD_BALANCE_FORK \ - | SD_SERIALIZE, \ - .last_balance = jiffies, \ - .balance_interval = 64, \ - .nr_balance_failed = 0, \ -} -#endif /* CONFIG_NUMA */ #ifdef CONFIG_SMP
arch/mips/include/asm/mach-ip27/topology.h +0 -17
@@ -36,23 +36,6 @@ extern unsigned char __node_distances[MAX_COMPACT_NODES][MAX_COMPACT_NODES];#define node_distance (from, to) (__node_distances[(from)][(to)]) -/* sched_domains SD_NODE_INIT for SGI IP27 machines */ -#define SD_NODE_INIT (struct sched_domain) { \ - .parent = NULL, \ - .child = NULL, \ - .groups = NULL, \ - .min_interval = 8, \ - .max_interval = 32, \ - .busy_factor = 32, \ - .imbalance_pct = 125, \ - .cache_nice_tries = 1, \ - .flags = SD_LOAD_BALANCE | \ - SD_BALANCE_EXEC, \ - .last_balance = jiffies, \ - .balance_interval = 1, \ - .nr_balance_failed = 0, \ -} -#include <asm-generic/topology.h> #endif /* _ASM_MACH_TOPOLOGY_H */
arch/powerpc/include/asm/topology.h +0 -36
@@ -18,12 +18,6 @@ struct device_node;*/ #define RECLAIM_DISTANCE 10 -/* -* Avoid creating an extra level of balancing (SD_ALLNODES) on the largest -* POWER7 boxes which have a maximum of 32 nodes. -*/ -#define SD_NODES_PER_DOMAIN 32 -#include <asm/mmzone.h> static inline int cpu_to_node (int cpu)
@@ -51,36 +45,6 @@ static inline int pcibus_to_node(struct pci_bus *bus) cpu_all_mask : \ cpumask_of_node(pcibus_to_node(bus))) -/* sched_domains SD_NODE_INIT for PPC64 machines */ -#define SD_NODE_INIT (struct sched_domain) { \ - .min_interval = 8, \ - .max_interval = 32, \ - .busy_factor = 32, \ - .imbalance_pct = 125, \ - .cache_nice_tries = 1, \ - .busy_idx = 3, \ - .idle_idx = 1, \ - .newidle_idx = 0, \ - .wake_idx = 0, \ - .forkexec_idx = 0, \ - \ - .flags = 1*SD_LOAD_BALANCE \ - | 0*SD_BALANCE_NEWIDLE \ - | 1*SD_BALANCE_EXEC \ - | 1*SD_BALANCE_FORK \ - | 0*SD_BALANCE_WAKE \ - | 1*SD_WAKE_AFFINE \ - | 0*SD_PREFER_LOCAL \ - | 0*SD_SHARE_CPUPOWER \ - | 0*SD_POWERSAVINGS_BALANCE \ - | 0*SD_SHARE_PKG_RESOURCES \ - | 1*SD_SERIALIZE \ - | 0*SD_PREFER_SIBLING \ - , \ - .last_balance = jiffies, \ - .balance_interval = 1, \ -} -extern int __node_distance (int ,int );#define node_distance (a, b) __node_distance(a, b)
arch/sh/include/asm/topology.h +0 -25
@@ -3,31 +3,6 @@#ifdef CONFIG_NUMA -/* sched_domains SD_NODE_INIT for sh machines */ -#define SD_NODE_INIT (struct sched_domain) { \ - .parent = NULL, \ - .child = NULL, \ - .groups = NULL, \ - .min_interval = 8, \ - .max_interval = 32, \ - .busy_factor = 32, \ - .imbalance_pct = 125, \ - .cache_nice_tries = 2, \ - .busy_idx = 3, \ - .idle_idx = 2, \ - .newidle_idx = 0, \ - .wake_idx = 0, \ - .forkexec_idx = 0, \ - .flags = SD_LOAD_BALANCE \ - | SD_BALANCE_FORK \ - | SD_BALANCE_EXEC \ - | SD_BALANCE_NEWIDLE \ - | SD_SERIALIZE, \ - .last_balance = jiffies, \ - .balance_interval = 1, \ - .nr_balance_failed = 0, \ -} -#define cpu_to_node (cpu) ((void)(cpu),0)#define parent_node (node) ((void)(node),0)
arch/sparc/include/asm/topology_64.h +0 -19
@@ -31,25 +31,6 @@ static inline int pcibus_to_node(struct pci_bus *pbus) cpu_all_mask : \ cpumask_of_node(pcibus_to_node(bus))) -#define SD_NODE_INIT (struct sched_domain) { \ - .min_interval = 8, \ - .max_interval = 32, \ - .busy_factor = 32, \ - .imbalance_pct = 125, \ - .cache_nice_tries = 2, \ - .busy_idx = 3, \ - .idle_idx = 2, \ - .newidle_idx = 0, \ - .wake_idx = 0, \ - .forkexec_idx = 0, \ - .flags = SD_LOAD_BALANCE \ - | SD_BALANCE_FORK \ - | SD_BALANCE_EXEC \ - | SD_SERIALIZE, \ - .last_balance = jiffies, \ - .balance_interval = 1, \ -} -#else /* CONFIG_NUMA */ #include <asm-generic/topology.h>
arch/tile/include/asm/topology.h +0 -26
@@ -78,32 +78,6 @@ static inline const struct cpumask *cpumask_of_node(int node) .balance_interval = 32, \ } -/* sched_domains SD_NODE_INIT for TILE architecture */ -#define SD_NODE_INIT (struct sched_domain) { \ - .min_interval = 16, \ - .max_interval = 512, \ - .busy_factor = 32, \ - .imbalance_pct = 125, \ - .cache_nice_tries = 1, \ - .busy_idx = 3, \ - .idle_idx = 1, \ - .newidle_idx = 2, \ - .wake_idx = 1, \ - .flags = 1*SD_LOAD_BALANCE \ - | 1*SD_BALANCE_NEWIDLE \ - | 1*SD_BALANCE_EXEC \ - | 1*SD_BALANCE_FORK \ - | 0*SD_BALANCE_WAKE \ - | 0*SD_WAKE_AFFINE \ - | 0*SD_PREFER_LOCAL \ - | 0*SD_SHARE_CPUPOWER \ - | 0*SD_SHARE_PKG_RESOURCES \ - | 1*SD_SERIALIZE \ - , \ - .last_balance = jiffies, \ - .balance_interval = 128, \ -} -/* By definition, we create nodes based on online memory. */ #define node_has_online_mem (nid) 1
arch/x86/include/asm/topology.h +0 -38
@@ -92,44 +92,6 @@ extern void setup_node_to_cpumask_map(void);#define pcibus_to_node (bus) __pcibus_to_node(bus) -#ifdef CONFIG_X86_32 -# define SD_CACHE_NICE_TRIES 1 -# define SD_IDLE_IDX 1 -#else -# define SD_CACHE_NICE_TRIES 2 -# define SD_IDLE_IDX 2 -#endif - -/* sched_domains SD_NODE_INIT for NUMA machines */ -#define SD_NODE_INIT (struct sched_domain) { \ - .min_interval = 8, \ - .max_interval = 32, \ - .busy_factor = 32, \ - .imbalance_pct = 125, \ - .cache_nice_tries = SD_CACHE_NICE_TRIES, \ - .busy_idx = 3, \ - .idle_idx = SD_IDLE_IDX, \ - .newidle_idx = 0, \ - .wake_idx = 0, \ - .forkexec_idx = 0, \ - \ - .flags = 1*SD_LOAD_BALANCE \ - | 1*SD_BALANCE_NEWIDLE \ - | 1*SD_BALANCE_EXEC \ - | 1*SD_BALANCE_FORK \ - | 0*SD_BALANCE_WAKE \ - | 1*SD_WAKE_AFFINE \ - | 0*SD_PREFER_LOCAL \ - | 0*SD_SHARE_CPUPOWER \ - | 0*SD_POWERSAVINGS_BALANCE \ - | 0*SD_SHARE_PKG_RESOURCES \ - | 1*SD_SERIALIZE \ - | 0*SD_PREFER_SIBLING \ - , \ - .last_balance = jiffies, \ - .balance_interval = 1, \ -} -extern int __node_distance (int ,int );#define node_distance (a, b) __node_distance(a, b)
arch/x86/kernel/process.c +8 -0
@@ -594,9 +594,17 @@ int mwait_usable(const struct cpuinfo_x86 *c) {u32 eax, ebx, ecx, edx; +/* Use mwait if idle=mwait boot option is given */ if (boot_option_idle_override ==IDLE_FORCE_MWAIT )return 1 ; +/* +* Any idle= boot option other than idle=mwait means that we must not +* use mwait. Eg: idle=halt or idle=poll or idle=nomwait +*/ +if (boot_option_idle_override !=IDLE_NO_OVERRIDE ) +return 0 ; +if (c->cpuid_level <MWAIT_INFO )return 0 ;
arch/x86/kernel/smpboot.c +69 -39
136 lines not on this page · raw message
arch/x86/mm/numa_emulation.c +6 -2
@@ -339,9 +339,11 @@ void __init numa_emulation(struct numa_meminfo *numa_meminfo, int numa_dist_cnt) }else {unsigned long n; - n =simple_strtoul (emu_cmdline,NULL ,0 ); + n =simple_strtoul (emu_cmdline, &emu_cmdline,0 ); ret =split_nodes_interleave (&ei, &pi,0 , max_addr, n); } +if (*emu_cmdline ==':' ) + emu_cmdline++;if (ret <0 ) goto no_emu;
@@ -418,7 +420,9 @@ void __init numa_emulation(struct numa_meminfo *numa_meminfo, int numa_dist_cnt)int physj = emu_nid_to_phys[j];int dist; -if (physi >= numa_dist_cnt || physj >= numa_dist_cnt) +if (get_option (&emu_cmdline, &dist) ==2 ) + ; +else if (physi >= numa_dist_cnt || physj >= numa_dist_cnt) dist = physi == physj ?LOCAL_DISTANCE :REMOTE_DISTANCE ;else
drivers/base/cpu.c +0 -4
@@ -330,8 +330,4 @@ void __init cpu_dev_init(void)panic ("Failed to register CPU subsystem" );cpu_dev_register_generic (); - -#if defined(CONFIG_SCHED_MC ) || defined(CONFIG_SCHED_SMT ) -sched_create_sysfs_power_savings_entries (cpu_subsys.dev_root); -#endif }
include/linux/cpu.h +0 -2
@@ -36,8 +36,6 @@ extern void cpu_remove_dev_attr(struct device_attribute *attr);extern int cpu_add_dev_attr_group (struct attribute_group *attrs);extern void cpu_remove_dev_attr_group (struct attribute_group *attrs); -extern int sched_create_sysfs_power_savings_entries (struct device *dev); -#ifdef CONFIG_HOTPLUG_CPU extern void unregister_cpu (struct cpu *cpu);extern ssize_t arch_cpu_probe (const char *,size_t );
include/linux/sched.h +1 -48
69 lines not on this page · raw message
include/linux/topology.h +0 -42
78 lines not on this page · raw message
kernel/sched/core.c +218 -202
503 lines not on this page · raw message
kernel/sched/debug.c +9 -3
@@ -202,7 +202,7 @@ void print_cfs_rq(struct seq_file *m, int cpu, struct cfs_rq *cfs_rq) SPLIT_NS(spread0));SEQ_printf (m," .%-30s: %d\n" ,"nr_spread_over" , cfs_rq->nr_spread_over); -SEQ_printf (m," .%-30s: %ld\n" ,"nr_running" , cfs_rq->nr_running); +SEQ_printf (m," .%-30s: %d\n" ,"nr_running" , cfs_rq->nr_running);SEQ_printf (m," .%-30s: %ld\n" ,"load" , cfs_rq->load.weight);#ifdef CONFIG_FAIR_GROUP_SCHED #ifdef CONFIG_SMP
@@ -260,8 +260,14 @@ static void print_cpu(struct seq_file *m, int cpu)SEQ_printf (m,"\ncpu#%d\n" , cpu);#endif -#define P (x) \ - SEQ_printf(m, " .%-30s: %Ld\n", #x, (long long)(rq->x)) +#define P (x) \ +do { \ + if (sizeof(rq->x) == 4) \ + SEQ_printf(m, " .%-30s: %ld\n", #x, (long)(rq->x)); \ + else \ + SEQ_printf(m, " .%-30s: %Ld\n", #x, (long long)(rq->x));\ +} while (0) +#define PN (x) \ SEQ_printf(m, " .%-30s: %Ld.%06ld\n", #x, SPLIT_NS(rq->x))
kernel/sched/fair.c +94 -368
836 lines not on this page · raw message
kernel/sched/idle_task.c +1 -1
@@ -4,7 +4,7 @@* idle-task scheduling class. * * (NOTE: these are not related to SCHED_IDLE tasks which are -* handled in sched_fair.c) +* handled in sched/fair.c) */ #ifdef CONFIG_SMP
kernel/sched/rt.c +26 -30
70 lines not on this page · raw message
kernel/sched/sched.h +4 -4
@@ -201,7 +201,7 @@ struct cfs_bandwidth { };/* CFS-related fields in a runqueue */ struct cfs_rq {struct load_weight load; -unsigned long nr_running, h_nr_running; +unsigned int nr_running, h_nr_running;u64 exec_clock;u64 min_vruntime;
@@ -279,7 +279,7 @@ static inline int rt_bandwidth_enabled(void)/* Real-Time classes' related field in a runqueue: */ struct rt_rq {struct rt_prio_array active; -unsigned long rt_nr_running; +unsigned int rt_nr_running; #if defined CONFIG_SMP || defined CONFIG_RT_GROUP_SCHED struct { int curr; /* highest queued rt task prio */
@@ -353,7 +353,7 @@ struct rq {* nr_running and cpu_load should be in the same cacheline because * remote CPUs use both these fields when doing load calculation. */ -unsigned long nr_running; +unsigned int nr_running;#define CPU_LOAD_IDX_MAX 5unsigned long cpu_load[CPU_LOAD_IDX_MAX ];unsigned long last_load_update_tick;
@@ -876,7 +876,7 @@ extern void resched_cpu(int cpu);extern struct rt_bandwidth def_rt_bandwidth;extern void init_rt_bandwidth (struct rt_bandwidth *rt_b,u64 period,u64 runtime); -extern void update_cpu_load (struct rq *this_rq); +extern void update_idle_cpu_load (struct rq *this_rq);#ifdef CONFIG_CGROUP_CPUACCT #include <linux/cgroup.h>
tools/power/cpupower/man/cpupower-set.1 +0 -9
@@ -85,15 +85,6 @@ Possible values are: savings .RE -sched_mc_power_savings is dependent upon SCHED_MC, which is -itself architecture dependent. - -sched_smt_power_savings is dependent upon SCHED_SMT, which -is itself architecture dependent. - -The two files are independent of each other. It is possible -that one file may be present without the other. - .SH "SEE ALSO" cpupower-info(1), cpupower-monitor(1), powertop(1) .PP
tools/power/cpupower/utils/helpers/sysfs.c +2 -33
@@ -362,22 +362,7 @@ char *sysfs_get_cpuidle_driver(void)*/ int sysfs_get_sched (const char *smt_mc) { -unsigned long value; -char linebuf[MAX_LINE_LEN ]; -char *endp; -char path[SYSFS_PATH_MAX ]; - -if (strcmp ("mc" , smt_mc) &&strcmp ("smt" , smt_mc)) -return -EINVAL ; - -snprintf (path,sizeof (path), -PATH_TO_CPU "sched_%s_power_savings" , smt_mc); -if (sysfs_read_file (path, linebuf,MAX_LINE_LEN ) ==0 ) -return -1 ; - value =strtoul (linebuf, &endp,0 ); -if (endp == linebuf || errno ==ERANGE ) -return -1 ; -return value; +return -ENODEV ; }/*
@@ -388,21 +373,5 @@ int sysfs_get_sched(const char *smt_mc)*/ int sysfs_set_sched (const char *smt_mc,int val) { -char linebuf[MAX_LINE_LEN ]; -char path[SYSFS_PATH_MAX ]; -struct stat statbuf; - -if (strcmp ("mc" , smt_mc) &&strcmp ("smt" , smt_mc)) -return -EINVAL ; - -snprintf (path,sizeof (path), -PATH_TO_CPU "sched_%s_power_savings" , smt_mc); -sprintf (linebuf,"%d" , val); - -if (stat (path, &statbuf) !=0 ) -return -ENODEV ; - -if (sysfs_write_file (path, linebuf,MAX_LINE_LEN ) ==0 ) -return -1 ; -return 0 ; +return -ENODEV ; }