Multiprocessor CPU scheduler patches
cover Multiprocessor CPU scheduler patches
Hi,
I hope that you can include the following set of CPU scheduler patches in -mm soon, if you have no other significant performance work going on.
There are some fairly significant changes, with a few basic aims:
* Improve SMT behaviour
* Improve CMP behaviour, CMP/NUMA scheduling (ie. Opteron)
* Reduce task movement, esp over NUMA nodes.
They are not going to be very well tuned for most usages at the moment (unfortunately dbt2/3-pgsql on OSDL isn't working, which is a good one). So hopefully I can address regressions as they come up.
There are a few problems with the scheduler currently:
Problem #1:
It has _very_ aggressive idle CPU pulling. Not only does it not really obey imbalances, it is also wrong for eg. an SMT CPU who's sibling is not idle. The reason this was done really is to bring down idle time on some workloads (dbt2-pgsql, other database stuff).
So I address this in the following ways; reduce special casing for idle balancing, revert some of the recent moves toward even more aggressive balancing.
Then provide a range of averaging levels for CPU "load averages", and we choose which to use in which situation on a sched-domain basis. This allows idle balancing to use a more instantaneous value for calculating load, so idle CPUs need not wait many timer ticks for the load averages to catch up. This can hopefully solve our idle time problems.
Also, further moderate "affine wakeups", which can tend to move most tasks to one CPU on some workloads and cause idle problems.
Problem #2:
The second problem is that balance-on-exec is not sched-domains aware. This means it will tend to (for example) fill up two cores of a CPU on one socket, then fill up two cores on the next socket, etc. What we want is to try to spread load evenly across memory controllers.
So make that sched-domains aware following the same pattern as find_busiest_group / find_busiest_queue.
Problem #3:
Lastly, implement balance-on-fork/clone again. I have come to the realisation that for NUMA, this is probably the best solution. Run-cloned-child-last has run out of steam on CMP systems. What it was supposed to do was provide a period where the child could be pulled to another CPU before it starts running and allocating memory. Unfortunately on CMP systems, this tends to just be to the other sibling.
Also, having such a difference between thread and process creation was not really ideal, so we balance on all types of fork/clone. This really helps some things (like STREAM) on CMP Opterons, but also hurts others, so naturally it is settable per-domain.
Problem #4:
Sched domains isn't very useful to me in its current form. Bring it up to date with what I've been using. I don't think anyone other than myself uses it so that should be OK.
Nick
PATCH 1/13 timestamp fixes
1/13
Some fixes for unsynchronised TSCs. A task's timestamp may have been set by another CPU. Although we try to adjust this correctly with the timestamp_last_tick field, there is no guarantee this will be exactly right.
kernel/sched.c +6 -1
@@ -648,6 +648,7 @@static void recalc_task_prio (task_t *p,unsigned long long now) { +/* Caller must always ensure 'now >= p->timestamp' */ unsigned long long __sleep_time = now - p->timestamp;unsigned long sleep_time;
@@ -2703,8 +2704,10 @@schedstat_inc (rq, sched_cnt); now =sched_clock (); -if (likely (now - prev->timestamp <NS_MAX_SLEEP_AVG )) +if (likely ((long long )now - prev->timestamp <NS_MAX_SLEEP_AVG )) run_time = now - prev->timestamp; +if (unlikely ((long long )now - prev->timestamp <0 )) + run_time =0 ;else run_time =NS_MAX_SLEEP_AVG ;
@@ -2782,6 +2785,8 @@if (!rt_task (next) && next->activated >0 ) {unsigned long long delta = now - next->timestamp; +if (unlikely ((long long )now - next->timestamp <0 )) + delta =0 ;if (next->activated ==1 ) delta = delta * (ON_RUNQUEUE_WEIGHT *128 /100 ) /128 ;
PATCH 2/13 improve pinned task handling
2/13
John Hawkes explained the problem best:
A large number of processes that are pinned to a single CPU results in every other CPU's load_balance() seeing this overloaded CPU as "busiest", yet move_tasks() never finds a task to pull-migrate. This condition occurs during module unload, but can also occur as a denial-of-service using sys_sched_setaffinity(). Several hundred CPUs performing this fruitless load_balance() will livelock on the busiest CPU's runqueue lock. A smaller number of CPUs will livelock if the pinned task count gets high.
Expanding slightly on John's patch, this one attempts to work out whether the balancing failure has been due to too many tasks pinned on the runqueue. This allows it to be basically invisible to the regular blancing paths (ie. when there are no pinned tasks). We can use this extra knowledge to shut down the balancing faster, and ensure the migration threads don't start running which is another problem observed in the wild.
kernel/sched.c +21 -10
@@ -1650,7 +1650,7 @@*/ static inline int can_migrate_task (task_t *p,runqueue_t *rq,int this_cpu, -struct sched_domain *sd,enum idle_type idle) +struct sched_domain *sd,enum idle_type idle,int *pinned) {/* * We do not migrate tasks that are:
@@ -1660,8 +1660,10 @@*/ if (task_running (rq, p))return 0 ; -if (!cpu_isset (this_cpu, p->cpus_allowed)) +if (!cpu_isset (this_cpu, p->cpus_allowed)) { + *pinned++;return 0 ; + }/* * Aggressive migration if:
@@ -1687,11 +1689,11 @@*/ static int move_tasks (runqueue_t *this_rq,int this_cpu,runqueue_t *busiest,unsigned long max_nr_move,struct sched_domain *sd, -enum idle_type idle) +enum idle_type idle,int *all_pinned) {prio_array_t *array, *dst_array;struct list_head *head, *curr; -int idx, pulled =0 ; +int idx, pulled =0 , pinned =0 ;task_t *tmp;if (max_nr_move <=0 || busiest->nr_running <=1 )
@@ -1735,7 +1737,7 @@ curr = curr->prev; -if (!can_migrate_task (tmp, busiest, this_cpu, sd, idle)) { +if (!can_migrate_task (tmp, busiest, this_cpu, sd, idle, &pinned)) {if (curr != head) goto skip_queue; idx++;
@@ -1761,6 +1763,9 @@ goto skip_bitmap; } out: + *all_pinned =0 ; +if (unlikely (pinned >= max_nr_move) && pulled ==0 ) + *all_pinned =1 ;return pulled; }
@@ -1935,7 +1940,7 @@struct sched_group *group;runqueue_t *busiest;unsigned long imbalance; -int nr_moved; +int nr_moved, all_pinned;spin_lock (&this_rq->lock);schedstat_inc (sd, lb_cnt[idle]);
@@ -1974,9 +1979,14 @@*/ double_lock_balance (this_rq, busiest); nr_moved =move_tasks (this_rq, this_cpu, busiest, - imbalance, sd, idle); + imbalance, sd, idle, + &all_pinned);spin_unlock (&busiest->lock); } +/* All tasks on this runqueue were pinned by CPU affinity */ +if (unlikely (all_pinned)) + goto out_balanced; +spin_unlock (&this_rq->lock);if (!nr_moved) {
@@ -2041,7 +2051,7 @@struct sched_group *group;runqueue_t *busiest =NULL ;unsigned long imbalance; -int nr_moved =0 ; +int nr_moved =0 , all_pinned;schedstat_inc (sd, lb_cnt[NEWLY_IDLE ]); group =find_busiest_group (sd, this_cpu, &imbalance,NEWLY_IDLE );
@@ -2061,7 +2071,7 @@schedstat_add (sd, lb_imbalance[NEWLY_IDLE ], imbalance); nr_moved =move_tasks (this_rq, this_cpu, busiest, - imbalance, sd,NEWLY_IDLE ); + imbalance, sd,NEWLY_IDLE , &all_pinned);if (!nr_moved)schedstat_inc (sd, lb_failed[NEWLY_IDLE ]);
@@ -2119,6 +2129,7 @@ cpu_group = sd->groups; do { for_each_cpu_mask(cpu, cpu_group->cpumask) { + int all_pinned; if (busiest_rq->nr_running <= 1) /* no more tasks left to move */ return;
@@ -2139,7 +2150,7 @@/* move a task from busiest_rq to target_rq */ double_lock_balance (busiest_rq, target_rq);if (move_tasks (target_rq, cpu, busiest_rq, -1 , sd,SCHED_IDLE )) { +1 , sd,SCHED_IDLE , &all_pinned)) {schedstat_inc (busiest_rq, alb_lost);schedstat_inc (target_rq, alb_gained); }else {
PATCH 3/13 rework schedstats
3/13
I have an updated userspace parser for this thing, if you are still keeping it on your website.
Move balancing fields into struct sched_domain, so we can get more useful results on systems with multiple domains (eg SMT+SMP, CMP+NUMA, SMP+NUMA, etc).
include/linux/sched.h +11 -2
@@ -462,17 +462,26 @@/* load_balance() stats */ unsigned long lb_cnt[MAX_IDLE_TYPES ];unsigned long lb_failed[MAX_IDLE_TYPES ]; +unsigned long lb_balanced[MAX_IDLE_TYPES ];unsigned long lb_imbalance[MAX_IDLE_TYPES ]; +unsigned long lb_gained[MAX_IDLE_TYPES ]; +unsigned long lb_hot_gained[MAX_IDLE_TYPES ];unsigned long lb_nobusyg[MAX_IDLE_TYPES ];unsigned long lb_nobusyq[MAX_IDLE_TYPES ]; +/* Active load balancing */ +unsigned long alb_cnt; +unsigned long alb_failed; +unsigned long alb_pushed; +/* sched_balance_exec() stats */ unsigned long sbe_attempts;unsigned long sbe_pushed;/* try_to_wake_up() stats */ -unsigned long ttwu_wake_affine; -unsigned long ttwu_wake_balance; +unsigned long ttwu_wake_remote; +unsigned long ttwu_move_affine; +unsigned long ttwu_move_balance;#endif };
kernel/sched.c +51 -63
267 lines not on this page · raw message
PATCH 4/13 find_busiest_group fixlets
4/13
Fix up a few small warts in the periodic multiprocessor rebalancing code.
kernel/sched.c +12 -11
@@ -1830,13 +1830,12 @@* by pulling tasks to us. Be careful of negative numbers as they'll * appear as very large values with unsigned longs. */ - *imbalance =min (max_load - avg_load, avg_load - this_load); -/* How much load to actually move to equalise the imbalance */ - *imbalance = (*imbalance *min (busiest->cpu_power, this->cpu_power)) - /SCHED_LOAD_SCALE ; + *imbalance =min ((max_load - avg_load) * busiest->cpu_power, + (avg_load - this_load) * this->cpu_power) + /SCHED_LOAD_SCALE ; -if (*imbalance <SCHED_LOAD_SCALE -1 ) { +if (*imbalance <SCHED_LOAD_SCALE ) {unsigned long pwr_now =0 , pwr_move =0 ;unsigned long tmp;
@@ -1862,14 +1861,16 @@ max_load - tmp);/* Amount of load we'd add */ - tmp =SCHED_LOAD_SCALE *SCHED_LOAD_SCALE /this->cpu_power; -if (max_load < tmp) - tmp = max_load; +if (max_load*busiest->cpu_power < +SCHED_LOAD_SCALE *SCHED_LOAD_SCALE ) + tmp = max_load*busiest->cpu_power/this->cpu_power; +else + tmp =SCHED_LOAD_SCALE *SCHED_LOAD_SCALE /this->cpu_power; pwr_move += this->cpu_power*min (SCHED_LOAD_SCALE , this_load + tmp); pwr_move /=SCHED_LOAD_SCALE ; -/* Move if we gain another 8th of a CPU worth of throughput */ -if (pwr_move < pwr_now +SCHED_LOAD_SCALE /8 ) +/* Move if we gain throughput */ +if (pwr_move <= pwr_now) goto out_balanced; *imbalance =1 ;
@@ -1877,7 +1878,7 @@ }/* Get rid of the scaling factor, rounding down as we divide */ - *imbalance = (*imbalance +1 ) /SCHED_LOAD_SCALE ; + *imbalance = *imbalance /SCHED_LOAD_SCALE ;return busiest;
PATCH 5/13 find_busiest_group cleanup
5/13
Cleanup find_busiest_group a bit. New sched-domains code means we can't have groups without a CPU.
kernel/sched.c +1 -5
@@ -1771,7 +1771,7 @@ do { unsigned long load; int local_group; - int i, nr_cpus = 0; + int i; local_group = cpu_isset(this_cpu, group->cpumask);
@@ -1785,13 +1785,9 @@else load =source_load (i); - nr_cpus++; avg_load += load; } -if (!nr_cpus) - goto nextgroup; - total_load += avg_load; total_pwr += group->cpu_power;
PATCH 6/13 no aggressive idle balancing
6/13
Remove the special casing for idle CPU balancing. Things like this are hurting for example on SMT, where are single sibling being idle doesn't really warrant a really aggressive pull over the NUMA domain, for example.
kernel/sched.c +0 -6
@@ -1875,15 +1875,9 @@/* Get rid of the scaling factor, rounding down as we divide */ *imbalance = *imbalance /SCHED_LOAD_SCALE ; -return busiest; out_balanced: -if (busiest && (idle ==NEWLY_IDLE || - (idle ==SCHED_IDLE && max_load >SCHED_LOAD_SCALE )) ) { - *imbalance =1 ; -return busiest; - } *imbalance =0 ;return NULL ;
PATCH 7/13 better active balancing heuristic
7/13
Fix up active load balancing a bit so it doesn't get called when it shouldn't. Reset the nr_balance_failed counter at more points where we have found conditions to be balanced. This reduces too aggressive active balancing seen on some workloads.
kernel/sched.c +10 -6
@@ -2009,6 +2009,7 @@schedstat_inc (sd, lb_balanced[idle]); + sd->nr_balance_failed =0 ;/* tune up the balancing interval */ if (sd->balance_interval < sd->max_interval) sd->balance_interval *=2 ;
@@ -2034,16 +2035,14 @@schedstat_inc (sd, lb_cnt[NEWLY_IDLE ]); group =find_busiest_group (sd, this_cpu, &imbalance,NEWLY_IDLE );if (!group) { -schedstat_inc (sd, lb_balanced[NEWLY_IDLE ]);schedstat_inc (sd, lb_nobusyg[NEWLY_IDLE ]); - goto out; + goto out_balanced; } busiest =find_busiest_queue (group);if (!busiest || busiest == this_rq) { -schedstat_inc (sd, lb_balanced[NEWLY_IDLE ]);schedstat_inc (sd, lb_nobusyq[NEWLY_IDLE ]); - goto out; + goto out_balanced; }/* Attempt to move tasks */
@@ -2054,11 +2053,16 @@ imbalance, sd, NEWLY_IDLE, &all_pinned);if (!nr_moved)schedstat_inc (sd, lb_failed[NEWLY_IDLE ]); +else + sd->nr_balance_failed =0 ;spin_unlock (&busiest->lock); - -out:return nr_moved; + +out_balanced: +schedstat_inc (sd, lb_balanced[NEWLY_IDLE ]); + sd->nr_balance_failed =0 ; +return 0 ; }/*
PATCH 8/13 generalised CPU load averaging
8/13
Do CPU load averaging over a number of different intervals. Allow each interval to be chosen by sending a parameter to source_load and target_load. 0 is instantaneous, idx > 0 returns a decaying average with the most recent sample weighted at 2^(idx-1). To a maximum of 3 (could be easily increased).
So generally a higher number will result in more conservative balancing.
include/asm-i386/topology.h +4 -0
@@ -77,6 +77,10 @@ .imbalance_pct = 125, \ .cache_hot_time = (10*1000000), \ .cache_nice_tries = 1, \ + .busy_idx = 3, \ + .idle_idx = 1, \ + .newidle_idx = 2, \ + .wake_idx = 1, \ .per_cpu_gain =100 , \ .flags = SD_LOAD_BALANCE \ |SD_BALANCE_EXEC \
include/asm-x86_64/topology.h +5 -1
@@ -49,7 +49,11 @@ .busy_factor = 32, \ .imbalance_pct = 125, \ .cache_hot_time = (10*1000000), \ - .cache_nice_tries = 1, \ + .cache_nice_tries = 2, \ + .busy_idx = 3, \ + .idle_idx = 2, \ + .newidle_idx = 1, \ + .wake_idx = 1, \ .per_cpu_gain =100 , \ .flags = SD_LOAD_BALANCE \ |SD_BALANCE_NEWIDLE \
include/linux/sched.h +4 -0
@@ -451,6 +451,10 @@unsigned long long cache_hot_time;/* Task considered cache hot (ns) */ unsigned int cache_nice_tries;/* Leave cache hot tasks for # tries */ unsigned int per_cpu_gain;/* CPU % gained by adding domain cpus */ +unsigned int busy_idx; +unsigned int idle_idx; +unsigned int newidle_idx; +unsigned int wake_idx;int flags;/* See SD_* */ /* Runtime fields. */
include/linux/topology.h +8 -0
@@ -86,6 +86,10 @@ .cache_hot_time = 0, \ .cache_nice_tries = 0, \ .per_cpu_gain = 25, \ + .busy_idx = 0, \ + .idle_idx = 0, \ + .newidle_idx = 0, \ + .wake_idx =0 , \ .flags = SD_LOAD_BALANCE \ |SD_BALANCE_NEWIDLE \ |SD_BALANCE_EXEC \
@@ -112,6 +116,10 @@ .cache_hot_time = (5*1000000/2), \ .cache_nice_tries = 1, \ .per_cpu_gain = 100, \ + .busy_idx = 2, \ + .idle_idx = 0, \ + .newidle_idx = 1, \ + .wake_idx =1 , \ .flags = SD_LOAD_BALANCE \ |SD_BALANCE_NEWIDLE \ |SD_BALANCE_EXEC \
kernel/sched.c +74 -64
252 lines not on this page · raw message
PATCH 9/13 less affine wakups
9/13
Do less affine wakeups. We're trying to reduce dbt2-pgsql idle time regressions here... make sure we don't don't move tasks the wrong way in an imbalance condition. Also, remove the cache coldness requirement from the calculation - this seems to induce sharp cutoff points where behaviour will suddenly change on some workloads if the load creeps slightly over or under some point. It is good for periodic balancing because in that case have otherwise have no other context to determine what task to move.
But also make a minor tweak to "wake balancing" - the imbalance tolerance is now set at half the domain's imbalance, so we get the opportunity to do wake balancing before the more random periodic rebalancing gets preformed.
kernel/sched.c +32 -25
@@ -1014,38 +1014,45 @@int idx = this_sd->wake_idx;unsigned int imbalance; + imbalance =100 + (this_sd->imbalance_pct -100 ) /2 ; + load =source_load (cpu, idx); this_load =target_load (this_cpu, idx); -/* -* If sync wakeup then subtract the (maximum possible) effect of -* the currently running task from the load of the current CPU: -*/ -if (sync) - this_load -=SCHED_LOAD_SCALE ; - -/* Don't pull the task off an idle CPU to a busy one */ -if (load <SCHED_LOAD_SCALE /2 && this_load >SCHED_LOAD_SCALE /2 ) - goto out_set_cpu; - new_cpu = this_cpu;/* Wake to this CPU if we can */ -if ((this_sd->flags &SD_WAKE_AFFINE ) && - !task_hot (p, rq->timestamp_last_tick, this_sd)) { -/* -* This domain has SD_WAKE_AFFINE and p is cache cold -* in this domain. -*/ -schedstat_inc (this_sd, ttwu_move_affine); - goto out_set_cpu; - }else if ((this_sd->flags &SD_WAKE_BALANCE ) && - imbalance*this_load <=100 *load) { +if (this_sd->flags &SD_WAKE_AFFINE ) { +unsigned long tl = this_load;/* -* This domain has SD_WAKE_BALANCE and there is -* an imbalance. +* If sync wakeup then subtract the (maximum possible) +* effect of the currently running task from the load +* of the current CPU: */ -schedstat_inc (this_sd, ttwu_move_balance); - goto out_set_cpu; +if (sync) + tl -=SCHED_LOAD_SCALE ; + +if ((tl <= load && + tl +target_load (cpu, idx) <=SCHED_LOAD_SCALE ) || +100 *(tl +SCHED_LOAD_SCALE ) <= imbalance*load) { +/* +* This domain has SD_WAKE_AFFINE and +* p is cache cold in this domain, and +* there is no bad imbalance. +*/ +schedstat_inc (this_sd, ttwu_move_affine); + goto out_set_cpu; + } + } + +/* +* Start passive balancing when half the imbalance_pct +* limit is reached. +*/ +if (this_sd->flags &SD_WAKE_BALANCE ) { +if (imbalance*this_load <=100 *load) { +schedstat_inc (this_sd, ttwu_move_balance); + goto out_set_cpu; + } } }
PATCH 10/13 remove aggressive idle balancing
10/13
Remove the very aggressive idle stuff that has recently gone into 2.6 - it is going against the direction we are trying to go. Hopefully we can regain performance through other methods.
include/asm-i386/topology.h +0 -1
@@ -85,7 +85,6 @@ .flags = SD_LOAD_BALANCE \ |SD_BALANCE_EXEC \ |SD_BALANCE_NEWIDLE \ - |SD_WAKE_IDLE \ | SD_WAKE_BALANCE, \ .last_balance = jiffies, \ .balance_interval = 1, \
include/asm-x86_64/topology.h +0 -1
@@ -58,7 +58,6 @@ .flags = SD_LOAD_BALANCE \ |SD_BALANCE_NEWIDLE \ |SD_BALANCE_EXEC \ - |SD_WAKE_IDLE \ | SD_WAKE_BALANCE, \ .last_balance = jiffies, \ .balance_interval = 1, \
include/linux/topology.h +0 -1
@@ -124,7 +124,6 @@ | SD_BALANCE_NEWIDLE \ |SD_BALANCE_EXEC \ |SD_WAKE_AFFINE \ - |SD_WAKE_IDLE \ | SD_WAKE_BALANCE, \ .last_balance = jiffies, \ .balance_interval = 1, \
kernel/sched.c +4 -21
@@ -412,22 +412,6 @@return rq; } -#ifdef CONFIG_SCHED_SMT -static int cpu_and_siblings_are_idle (int cpu) -{ -int sib; -for_each_cpu_mask (sib, cpu_sibling_map[cpu]) { -if (idle_cpu (sib)) -continue ; -return 0 ; - } - -return 1 ; -} -#else -#define cpu_and_siblings_are_idle (A ) idle_cpu(A) -#endif -#ifdef CONFIG_SCHEDSTATS /* * Called when a process is dequeued from the active array and given
@@ -1650,16 +1634,15 @@/* * Aggressive migration if: -* 1) the [whole] cpu is idle, or +* 1) task is cache cold, or * 2) too many balance attempts have failed. */ -if (cpu_and_siblings_are_idle (this_cpu) || \ - sd->nr_balance_failed > sd->cache_nice_tries) +if (sd->nr_balance_failed > sd->cache_nice_tries)return 1 ;if (task_hot (p, rq->timestamp_last_tick, sd)) -return 0 ; +return 0 ;return 1 ; }
@@ -2131,7 +2114,7 @@if (cpu_isset (cpu, visited_cpus))continue ;cpu_set (cpu, visited_cpus); -if (!cpu_and_siblings_are_idle (cpu) || cpu == busiest_cpu) +if (cpu == busiest_cpu)continue ; target_rq =cpu_rq (cpu);
PATCH 11/13 sched-domains aware balance-on-fork
11/13
Reimplement the balance on exec balancing to be sched-domains aware. Use this to also do balance on fork balancing. Make x86_64 do balance on fork over the NUMA domain.
The problem that the non sched domains aware blancing became apparent on dual core, multi socket opterons. What we want is for the new tasks to be sent to a different socket, but more often than not, we would first load up our sibling core, or fill two cores of a single remote socket before selecting a new one.
This gives large improvements to STREAM on such systems.
include/asm-x86_64/topology.h +2 -0
@@ -54,9 +54,11 @@ .idle_idx = 2, \ .newidle_idx = 1, \ .wake_idx = 1, \ + .forkexec_idx = 1, \ .per_cpu_gain = 100, \ .flags =SD_LOAD_BALANCE \ |SD_BALANCE_NEWIDLE \ + |SD_BALANCE_FORK \ |SD_BALANCE_EXEC \ | SD_WAKE_BALANCE, \ .last_balance = jiffies, \
include/linux/sched.h +6 -4
@@ -423,10 +423,11 @@#define SD_LOAD_BALANCE 1/* Do load balancing on this domain. */ #define SD_BALANCE_NEWIDLE 2/* Balance when about to become idle */ #define SD_BALANCE_EXEC 4/* Balance on exec */ -#define SD_WAKE_IDLE 8/* Wake to idle CPU on task wakeup */ -#define SD_WAKE_AFFINE 16/* Wake task to waking CPU */ -#define SD_WAKE_BALANCE 32/* Perform balancing at task wakeup */ -#define SD_SHARE_CPUPOWER 64/* Domain members share cpu power */ +#define SD_BALANCE_FORK 8/* Balance on fork, clone */ +#define SD_WAKE_IDLE 16/* Wake to idle CPU on task wakeup */ +#define SD_WAKE_AFFINE 32/* Wake task to waking CPU */ +#define SD_WAKE_BALANCE 64/* Perform balancing at task wakeup */ +#define SD_SHARE_CPUPOWER 128/* Domain members share cpu power */ struct sched_group {struct sched_group *next;/* Must be a circular list */
@@ -455,6 +456,7 @@unsigned int idle_idx;unsigned int newidle_idx;unsigned int wake_idx; +unsigned int forkexec_idx;int flags;/* See SD_* */ /* Runtime fields. */
include/linux/topology.h +2 -0
@@ -90,6 +90,7 @@ .idle_idx = 0, \ .newidle_idx = 0, \ .wake_idx = 0, \ + .forkexec_idx =0 , \ .flags = SD_LOAD_BALANCE \ |SD_BALANCE_NEWIDLE \ |SD_BALANCE_EXEC \
@@ -120,6 +121,7 @@ .idle_idx = 0, \ .newidle_idx = 1, \ .wake_idx = 1, \ + .forkexec_idx =0 , \ .flags = SD_LOAD_BALANCE \ |SD_BALANCE_NEWIDLE \ |SD_BALANCE_EXEC \
kernel/sched.c +109 -55
209 lines not on this page · raw message
PATCH 12/13 schedstats additions for sched-balance-fork
12/13
Add SCHEDSTAT statistics for sched-balance-fork.
include/linux/sched.h +8 -2
@@ -480,10 +480,16 @@unsigned long alb_failed;unsigned long alb_pushed; -/* sched_balance_exec() stats */ -unsigned long sbe_attempts; +/* SD_BALANCE_EXEC stats */ +unsigned long sbe_cnt; +unsigned long sbe_balanced;unsigned long sbe_pushed; +/* SD_BALANCE_FORK stats */ +unsigned long sbf_cnt; +unsigned long sbf_balanced; +unsigned long sbf_pushed; +/* try_to_wake_up() stats */ unsigned long ttwu_wake_remote;unsigned long ttwu_move_affine;
kernel/sched.c +36 -27
106 lines not on this page · raw message
PATCH 13/13 basic tuning
13/13
Do some basic initial tuning.
include/asm-x86_64/topology.h +1 -2
@@ -52,12 +52,11 @@ .cache_nice_tries = 2, \ .busy_idx = 3, \ .idle_idx = 2, \ - .newidle_idx = 1, \ + .newidle_idx = 0, \ .wake_idx = 1, \ .forkexec_idx = 1, \ .per_cpu_gain = 100, \ .flags =SD_LOAD_BALANCE \ - |SD_BALANCE_NEWIDLE \ |SD_BALANCE_FORK \ |SD_BALANCE_EXEC \ | SD_WAKE_BALANCE, \
include/linux/topology.h +4 -5
@@ -118,15 +118,14 @@ .cache_nice_tries = 1, \ .per_cpu_gain = 100, \ .busy_idx = 2, \ - .idle_idx = 0, \ - .newidle_idx = 1, \ + .idle_idx = 1, \ + .newidle_idx = 2, \ .wake_idx = 1, \ - .forkexec_idx = 0, \ + .forkexec_idx = 1, \ .flags =SD_LOAD_BALANCE \ |SD_BALANCE_NEWIDLE \ |SD_BALANCE_EXEC \ - |SD_WAKE_AFFINE \ - | SD_WAKE_BALANCE, \ + | SD_WAKE_AFFINE, \ .last_balance = jiffies, \ .balance_interval = 1, \ .nr_balance_failed = 0, \
Re: [PATCH 12/13] schedstats additions for sched-balance-fork
In reply to Nick PigginQuoted 3 lines · Nick Piggin
[PATCH 11/13] sched-domains aware balance-on-fork
[PATCH 12/13] schedstats additions for sched-balance-fork
[PATCH 13/13] basic tuning
STREAMS numbers tricky. It's pretty much the only benchmark that 1) relies on being able to allocate alot of RAM in a NUMA-friendly way 2) does all of its memory allocation in the first timeslice of cloned worker threads.
There is little help we get from userspace, and i'm not sure we want to add scheduler overhead for this single benchmark - when something like a _tiny_ bit of NUMAlib use within the OpenMP library would probably solve things equally well!
Anyway, the code itself looks fine and it would be good if it improved things, so:
but this too needs alot of testing, and it the one that has the highest likelyhood of actually not making it upstream.
Ingo
Re: [PATCH 12/13] schedstats additions for sched-balance-fork
Quoted 8 lines · Ingo Molnar
Quoted 3 lines · Nick Piggin
[PATCH 11/13] sched-domains aware balance-on-fork
[PATCH 12/13] schedstats additions for sched-balance-fork
[PATCH 13/13] basic tuning
Quoted 5 lines
STREAMS numbers tricky. It's pretty much the only benchmark that 1) relies on being able to allocate alot of RAM in a NUMA-friendly way 2) does all of its memory allocation in the first timeslice of cloned worker threads.
I know what you mean... but this is not _just_ for STREAM. Firstly, if we start 4 tasks on one core (of a 4 socket / 8 core system), and just let them be moved around by the periodic balancer, they will tend to cluster on 2 or 3 CPUs, and that will be the steady state.
Quoted 5 lines
There is little help we get from userspace, and i'm not sure we want to add scheduler overhead for this single benchmark - when something like a _tiny_ bit of NUMAlib use within the OpenMP library would probably solve things equally well!
True, for OpenMP apps (and this work shouldn't stop that from happening). But other threaded apps are also important, and fork()ed apps can be important too.
What I hear from the NUMA guys (POWER5, AMD) is that they really want to keep memory controllers busy. This seems to be the best way to do it.
There are a few differences between this and when we last tried it. The main thing is that the balancer is now sched-domains aware. I hope we can get it to do the right thing more often (at least it is a per domain flag, so those who don't want it don't turn it on).
Quoted 8 lines
Anyway, the code itself looks fine and it would be good if it improved things, so:
Acked-by Ingo Molnar <[email protected]>but this too needs alot of testing, and it the one that has the highest likelyhood of actually not making it upstream.
Thanks for reviewing.
Re: [PATCH 12/13] schedstats additions for sched-balance-fork
In reply to Ingo MolnarThere is little help we get from userspace, and i'm not sure we want to add scheduler overhead for this single benchmark - when something like a _tiny_ bit of NUMAlib use within the OpenMP library would probably solve things equally well!
There's has been a general problem with sched domains and it trying to meet two goals: "1) spread things around evenly within a domain and balance across domains infrequently", and "2) load up cores before loading up siblings, even at the expense of violating 1)".
We've had trouble getting both 1) and 2) implemented correctly in the past. If this patch gets us closer to that nirvana, it will be valuable regardless of the benchmark it also happens to be improving.
Regardless, I agree it will need good testing, and we may need to pick the wheat from the chaff.
Rick
Re: [PATCH 12/13] schedstats additions for sched-balance-fork
Quoted 10 lines · Rick Lindsley
There is little help we get from userspace, and i'm not sure we want to add scheduler overhead for this single benchmark - when something like a _tiny_ bit of NUMAlib use within the OpenMP library would probably solve things equally well!
There's has been a general problem with sched domains and it trying to meet two goals: "1) spread things around evenly within a domain and balance across domains infrequently", and "2) load up cores before loading up siblings, even at the expense of violating 1)".
Yes, you hit the nail on the head. Well, the other (potentially more problematic) part of the equation is "3) keep me close to my parent and siblings, because we'll be likely to share memory and/or communicate".
However: I'm hoping that on unloaded or lightly loaded NUMA systems, it is usually the right choice to spread tasks across nodes. Especially on the newer breed of low remote latency, high bandwidth systems like Opterons and POWER5s.
When load ramps up a bit and we start saturating CPUs, the amount of balance-on-forking should slow down, so we start to fulfil requirement 3 for workloads that perhaps resemble more general purpose server stuff.
That's the plan anyway. We'll see...
Re: [PATCH 10/13] remove aggressive idle balancing
In reply to Nick PigginQuoted 5 lines · Nick Piggin
[PATCH 6/13] no aggressive idle balancing
[PATCH 8/13] generalised CPU load averaging
[PATCH 9/13] less affine wakups
[PATCH 10/13] remove aggressive idle balancing
they look fine, but these are the really scary ones :-) Maybe we could do #8 and #9 first, then #6+#10. But it's probably pointless to look at these in isolation.
Ingo
Re: [PATCH 10/13] remove aggressive idle balancing
Quoted 5 lines · Ingo Molnar
Quoted 2 lines · Nick Piggin
[PATCH 6/13] no aggressive idle balancing
Quoted 9 lines
Quoted 3 lines
[PATCH 8/13] generalised CPU load averaging
[PATCH 9/13] less affine wakups
[PATCH 10/13] remove aggressive idle balancingthey look fine, but these are the really scary ones :-) Maybe we could do #8 and #9 first, then #6+#10. But it's probably pointless to look at these in isolation.
Oh yes, they are very scary and I guarantee they'll cause problems :P
I didn't have any plans to get these in for 2.6.12 (2.6.13 at the very earliest). But it will be nice if Andrew can pick these up early so we try to get as much regression testing as possible.
I pretty much agree with your ealier breakdown of the patches (ie. some are fixes, others fairly straightfoward improvements that may get into 2.6.12, of course). Thanks very much for the review.
I expect to rework the patches, and things will get tuned and changed around a bit... Any problem with you taking these now though Andrew?
Nick
Re: [PATCH 10/13] remove aggressive idle balancing
Quoted 11 lines · Nick Piggin
Quoted 10 lines · Ingo Molnar
Quoted 5 lines · Nick Piggin
[PATCH 6/13] no aggressive idle balancing
[PATCH 8/13] generalised CPU load averaging
[PATCH 9/13] less affine wakups
[PATCH 10/13] remove aggressive idle balancing
Quoted 7 lines
Quoted 4 lines
they look fine, but these are the really scary ones :-) Maybe we could do #8 and #9 first, then #6+#10. But it's probably pointless to look at these in isolation.
Oh yes, they are very scary and I guarantee they'll cause problems :P
:-|
Quoted 10 lines
I didn't have any plans to get these in for 2.6.12 (2.6.13 at the very earliest). But it will be nice if Andrew can pick these up early so we try to get as much regression testing as possible.
I pretty much agree with your ealier breakdown of the patches (ie. some are fixes, others fairly straightfoward improvements that may get into 2.6.12, of course). Thanks very much for the review.
I expect to rework the patches, and things will get tuned and changed around a bit... Any problem with you taking these now though Andrew?
sure, fine with me.
Ingo
Re: [PATCH 10/13] remove aggressive idle balancing
In reply to Nick PigginQuoted 11 lines · Nick Piggin
Quoted 10 lines · Ingo Molnar
Quoted 5 lines · Nick Piggin
[PATCH 6/13] no aggressive idle balancing
[PATCH 8/13] generalised CPU load averaging
[PATCH 9/13] less affine wakups
[PATCH 10/13] remove aggressive idle balancing
Quoted 7 lines
Quoted 4 lines
they look fine, but these are the really scary ones :-) Maybe we could do #8 and #9 first, then #6+#10. But it's probably pointless to look at these in isolation.
Oh yes, they are very scary and I guarantee they'll cause problems :P
By code inspection, I see an issue with this patch
[PATCH 10/13] remove aggressive idle balancing
Why are we removing cpu_and_siblings_are_idle check from active_load_balance? In case of SMT, we want to give prioritization to an idle package while doing active_load_balance(infact, active_load_balance will be kicked mainly because there is an idle package)
Just the re-addition of cpu_and_siblings_are_idle check to active_load_balance might not be enough. We somehow need to communicate this to move_tasks, otherwise can_migrate_task will fail and we will never be able to do active_load_balance.
thanks,
suresh
Re: [PATCH 10/13] remove aggressive idle balancing
Quoted 11 lines · Suresh B
By code inspection, I see an issue with this patch
[PATCH 10/13] remove aggressive idle balancingWhy are we removing
cpu_and_siblings_are_idlecheck fromactive_load_balance? In case of SMT, we want to give prioritization to an idle package while doingactive_load_balance(infact, active_load_balance will be kicked mainly because there is an idle package)Just the re-addition of
cpu_and_siblings_are_idlecheck toactive_load_balancemight not be enough. We somehow need to communicate this tomove_tasks, otherwisecan_migrate_taskwill fail and we will never be able to doactive_load_balance.
Active balancing should only kick in after the prescribed number of rebalancing failures - can_migrate_task will see this, and will allow the balancing to take place.
That said, we currently aren't doing _really_ well for SMT on some workloads, however with this patch we are heading in the right direction I think.
I have been mainly looking at tuning CMP Opterons recently (they are closer to a "traditional" SMP+NUMA than SMT, when it comes to the scheduler's point of view). However, in earlier revisions of the patch I had been looking at SMT performance and was able to get it much closer to perfect:
I was working on a 4 socket x440 with HT. The problem area is usually when the load is lower than the number of logical CPUs. So on tbench, we do say 450MB/s with 4 or more threads without HT, and 550MB/s with 8 or more threads with HT, however we only do 300MB/s with 4 threads.
Those aren't the exact numbers, but that's basically what they look like. Now I was able to bring the 4 thread + HT case much closer to the 4 thread - HT numbers, but with earlier patchsets. When I get a chance I will do more tests on the HT system, but the x440 is infuriating for fine tuning performance, because it is a NUMA system, but it doesn't tell the kernel about it, so it will randomly schedule things on "far away" CPUs, and results vary.
PS. Another thing I would like to see tested is a 3 level domain setup (SMT + SMP + NUMA). I don't have access to one though.
Re: [PATCH 10/13] remove aggressive idle balancing
Nick,
Quoted 17 lines · Nick Piggin
Quoted 9 lines · Suresh B
By code inspection, I see an issue with this patch
[PATCH 10/13] remove aggressive idle balancingWhy are we removing
cpu_and_siblings_are_idlecheck fromactive_load_balance? In case of SMT, we want to give prioritization to an idle package while doingactive_load_balance(infact, active_load_balance will be kicked mainly because there is an idle package)Quoted 5 lines
Just the re-addition of
cpu_and_siblings_are_idlecheck toactive_load_balancemight not be enough. We somehow need to communicate this tomove_tasks, otherwisecan_migrate_taskwill fail and we will never be able to doactive_load_balance.
Quoted 3 lines
Active balancing should only kick in after the prescribed number of rebalancing failures -
can_migrate_taskwill see this, and will allow the balancing to take place.
We are resetting the nr_balance_failed to cache_nice_tries after kicking active balancing. But can_migrate_task will succeed only if nr_balance_failed > cache_nice_tries.
Quoted 4 lines
That said, we currently aren't doing _really_ well for SMT on some workloads, however with this patch we are heading in the right direction I think.
Lets take an example of three packages with two logical threads each. Assume P0 is loaded with two processes(one in each logical thread), P1 contains only one process and P2 is idle.
In this example, active balance will be kicked on one of the threads(assume thread 0) in P0, which then should find an idle package and move it to one of the idle threads in P2.
With your current patch, idle package check in active_load_balance has disappeared, and we may endup moving the process from thread 0 to thread 1 in P0. I can't really make logic out of the active_load_balance code after your patch 10/13
Quoted 7 lines
I have been mainly looking at tuning CMP Opterons recently (they are closer to a "traditional" SMP+NUMA than SMT, when it comes to the scheduler's point of view). However, in earlier revisions of the patch I had been looking at SMT performance and was able to get it much closer to perfect:
I am reasonably sure that the removal of cpu_and_siblings_are_idle check from active_load_balance will cause HT performance regressions.
Quoted 5 lines
I was working on a 4 socket x440 with HT. The problem area is usually when the load is lower than the number of logical CPUs. So on tbench, we do say 450MB/s with 4 or more threads without HT, and 550MB/s with 8 or more threads with HT, however we only do 300MB/s with 4 threads.
Are you saying 2.6.11 has this problem?
Quoted 9 lines
Those aren't the exact numbers, but that's basically what they look like. Now I was able to bring the 4 thread + HT case much closer to the 4 thread - HT numbers, but with earlier patchsets. When I get a chance I will do more tests on the HT system, but the x440 is infuriating for fine tuning performance, because it is a NUMA system, but it doesn't tell the kernel about it, so it will randomly schedule things on "far away" CPUs, and results vary.
Why don't you use any other simple HT+SMP system?
I will also do some performance analysis with your other patches on some of the systems that I have access to.
thanks,
suresh
Re: [PATCH 10/13] remove aggressive idle balancing
Quoted 5 lines · Suresh B
Nick,
Quoted 1 line · Nick Piggin
Quoted 8 lines
Quoted 3 lines
Active balancing should only kick in after the prescribed number of rebalancing failures -
can_migrate_taskwill see this, and will allow the balancing to take place.We are resetting the
nr_balance_failedtocache_nice_triesafter kicking active balancing. Butcan_migrate_taskwill succeed only ifnr_balance_failed>cache_nice_tries.
It is indeed, thanks for catching that. We should probably make it reset the count to the point where it will start moving cache hot tasks (ie. cache_nice_tries+1).
I'll look at that and send Andrew a patch.
Quoted 6 lines
Quoted 4 lines
That said, we currently aren't doing _really_ well for SMT on some workloads, however with this patch we are heading in the right direction I think.
Quoted 7 lines
Lets take an example of three packages with two logical threads each. Assume P0 is loaded with two processes(one in each logical thread), P1 contains only one process and P2 is idle.
In this example, active balance will be kicked on one of the threads(assume
thread 0) in P0, which then should find an idle package and move it to one of the idle threads in P2.With your current patch, idle package check in
active_load_balancehas disappeared, and we may endup moving the process from thread 0 to thread 1 in P0. I can't really make logic out of theactive_load_balancecode after your patch 10/13
Ah yep, right you are there, too. I obviously hadn't looked closely enough at the recent active_load_balance patches that had gone in :( What should probably do is heed the "push_cpu" prescription (push_cpu is now unused).
I think active_load_balance is too complex at the moment, but still too dumb to really make the right choice here over the full range of domains. What we need to do is pass in some more info from load_balance, so active_load_balance doesn't need any "smarts".
Thanks for pointing this out too. I'll make a patch.
Quoted 9 lines
Quoted 7 lines
I have been mainly looking at tuning CMP Opterons recently (they are closer to a "traditional" SMP+NUMA than SMT, when it comes to the scheduler's point of view). However, in earlier revisions of the patch I had been looking at SMT performance and was able to get it much closer to perfect:
Quoted 3 lines
I am reasonably sure that the removal of
cpu_and_siblings_are_idlecheck fromactive_load_balancewill cause HT performance regressions.
Yep.
Quoted 10 lines
Quoted 6 lines
I was working on a 4 socket x440 with HT. The problem area is usually when the load is lower than the number of logical CPUs. So on tbench, we do say 450MB/s with 4 or more threads without HT, and 550MB/s with 8 or more threads with HT, however we only do 300MB/s with 4 threads.
Are you saying 2.6.11 has this problem?
I think so. I'll have a look at it again.
Quoted 11 lines
Quoted 9 lines
Those aren't the exact numbers, but that's basically what they look like. Now I was able to bring the 4 thread + HT case much closer to the 4 thread - HT numbers, but with earlier patchsets. When I get a chance I will do more tests on the HT system, but the x440 is infuriating for fine tuning performance, because it is a NUMA system, but it doesn't tell the kernel about it, so it will randomly schedule things on "far away" CPUs, and results vary.
Quoted 2 lines
Why don't you use any other simple HT+SMP system?
Yes I will, of course. Some issues can become more pronounced with more physical CPUs, but the main reason is that the x440 is the only one with HT at work where I was doing testing.
To be honest I hadn't looked hard enough at the HT issues yet as you've noticed. So thanks for the review and I'll fix things up.
Quoted 3 lines
I will also do some performance analysis with your other patches on some of the systems that I have access to.
Thanks.
Nick
Re: [PATCH 10/13] remove aggressive idle balancing
Nick,
Quoted 9 lines · Nick Piggin
Quoted 4 lines · Suresh B
We are resetting the
nr_balance_failedtocache_nice_triesafter kicking active balancing. Butcan_migrate_taskwill succeed only ifnr_balance_failed>cache_nice_tries.It is indeed, thanks for catching that. We should probably make it reset the count to the point where it will start moving cache hot tasks (ie. cache_nice_tries+1).
That still might not be enough. We probably need to pass push_cpu's sd to move_tasks call in active_load_balance, instead of current busiest_cpu's sd. Just like push_cpu, we need to add one more field to the runqueue which will specify the domain level of the push_cpu at which we have an imbalance.
Quoted 23 lines
I'll look at that and send Andrew a patch.
Quoted 10 lines
Quoted 4 lines
That said, we currently aren't doing _really_ well for SMT on some workloads, however with this patch we are heading in the right direction I think.
Lets take an example of three packages with two logical threads each. Assume P0 is loaded with two processes(one in each logical thread), P1 contains only one process and P2 is idle.
Quoted 9 lines
In this example, active balance will be kicked on one of the threads(assume thread 0) in P0, which then should find an idle package and move it to one of the idle threads in P2.
With your current patch, idle package check in
active_load_balancehas disappeared, and we may endup moving the process from thread 0 to thread 1 in P0. I can't really make logic out of theactive_load_balancecode after your patch 10/13
Quoted 4 lines
Ah yep, right you are there, too. I obviously hadn't looked closely enough at the recent
active_load_balancepatches that had gone in :( What should probably do is heed the "push_cpu" prescription (push_cpuis now unused).
push_cpu might not be the ideal destination in all cases. Take a NUMA domain above SMT+SMP domains in my above example. Assume P0, P1 is in node-0 and P2, P3 in node-1. Assume Loads of P0,P1,P2 are same as the above example,with P3 containing one process load. Now any idle thread in P2 or P3 can trigger active load balance on P0. We should be selecting thread in P2 ideally (currently this is what we get with idle package check). But with push_cpu, we might move to the idle thread in P3 and then finally move to P2(it will be a two step process)
thanks,
suresh
Re: [PATCH 10/13] remove aggressive idle balancing
Quoted 10 lines · Suresh B
Nick,
Quoted 6 lines · Nick Piggin
Quoted 3 lines · Suresh B
We are resetting the
nr_balance_failedtocache_nice_triesafter kicking active balancing. Butcan_migrate_taskwill succeed only ifnr_balance_failed>cache_nice_tries.
Quoted 9 lines
Quoted 3 lines
It is indeed, thanks for catching that. We should probably make it reset the count to the point where it will start moving cache hot tasks (ie. cache_nice_tries+1).
That still might not be enough. We probably need to pass push_cpu's
sd tomove_taskscall inactive_load_balance, instead of current busiest_cpu's
sd. Just likepush_cpu, we need to add one more field to the runqueue which will specify the domain level of thepush_cpuat which we have an imbalance.
It should be the lowest domain level that spans both this_cpu and push_cpu, and has the SD_BALANCE flag set. We could possibly be a bit more general here, but so long as nobody is coming up with weird and wonderful sched_domains schemes, push_cpu should give you all the info needed.
Quoted 6 lines
Quoted 4 lines
Ah yep, right you are there, too. I obviously hadn't looked closely enough at the recent
active_load_balancepatches that had gone in :( What should probably do is heed the "push_cpu" prescription (push_cpuis now unused).
Quoted 9 lines
push_cpumight not be the ideal destination in all cases. Take a NUMA domain above SMT+SMP domains in my above example. Assume P0, P1 is in node-0 and P2, P3 in node-1. Assume Loads of P0,P1,P2 are same as the above example,with P3 containing one process load. Now any idle thread in P2 or P3 can trigger active load balance on P0. We should be selecting thread in P2 ideally (currently this is what we get with idle package check). But withpush_cpu, we might move to the idle thread in P3 and then finally move to P2(it will be a two step process)
Hmm yeah. It is a bit tricky. We don't currently do exceptionally well at this sort of "balancing over multiple domains" very well in the periodic balancer either.
But at this stage I prefer to not get overly complex, and allow some imperfect task movement, because it should rarely be a problem, and is much better than it was before. The main place where it can go wrong is multi-level NUMA balancing, where moving a task twice (between different nodes) can cause more problems.
Nick
Re: [PATCH 10/13] remove aggressive idle balancing
Quoted 9 lines · Nick Piggin
Quoted 5 lines · Suresh B
That still might not be enough. We probably need to pass push_cpu's sd to
move_taskscall inactive_load_balance, instead of current busiest_cpu's sd. Just likepush_cpu, we need to add one more field to the runqueue which will specify the domain level of thepush_cpuat which we have an imbalance.It should be the lowest domain level that spans both
this_cpuandpush_cpu, and has theSD_BALANCEflag set. We could possibly be a bit
I agree.
Quoted 14 lines
more general here, but so long as nobody is coming up with weird and wonderful
sched_domainsschemes,push_cpushould give you all the info needed.Quoted 9 lines
push_cpumight not be the ideal destination in all cases. Take a NUMA domain above SMT+SMP domains in my above example. Assume P0, P1 is in node-0 and P2, P3 in node-1. Assume Loads of P0,P1,P2 are same as the above example,with P3 containing one process load. Now any idle thread in P2 or P3 can trigger active load balance on P0. We should be selecting thread in P2 ideally (currently this is what we get with idle package check). But withpush_cpu, we might move to the idle thread in P3 and then finally move to P2(it will be a two step process)
Quoted 3 lines
Hmm yeah. It is a bit tricky. We don't currently do exceptionally well at this sort of "balancing over multiple domains" very well in the periodic balancer either.
With periodic balancer, moved tasks will not be actively running and by the time it gets a cpu slot, it will most probably be on the correct cpu (though "most probably" is the key word here ;-)
Quoted 5 lines
But at this stage I prefer to not get overly complex, and allow some imperfect task movement, because it should rarely be a problem, and is much better than it was before. The main place where it can go wrong is multi-level NUMA balancing, where moving a task twice (between different nodes) can cause more problems.
With active_load_balance, we will be moving the currently running process. So obviously if we can move in one single step, that will be nice.
I agree with the complexity part though. And it will become more complex with upcoming dual-core requirements.
thanks,
suresh
Re: [PATCH 7/13] better active balancing heuristic
In reply to Nick PigginQuoted 1 line · Nick Piggin
7/13
yeah, we need this one too.
Ingo
Re: [PATCH 4/13] find_busiest_group fixlets
In reply to Nick PigginQuoted 2 lines · Nick Piggin
4/13
5/13
#insert <previous mail>
Ingo
Re: [PATCH 3/13] rework schedstats
In reply to Nick PigginQuoted 4 lines · Nick Piggin
3/13
I have an updated userspace parser for this thing, if you are still keeping it on your website.
Ingo
Re: [PATCH 3/13] rework schedstats
In reply to Nick PigginI have an updated userspace parser for this thing, if you are still keeping it on your website.
Sure, be happy to include it, thanks! Send it along. Is it for version 11 or version 12?
Move balancing fields into struct sched_domain, so we can get more useful results on systems with multiple domains (eg SMT+SMP, CMP+NUMA, SMP+NUMA, etc).
It looks like you've also removed the stats for pull_task() and wake_up_new_task(). Was this intentional, or accidental?
I can't quite get the patch to apply cleanly against several variants of 2.6.10 or 2.6.11-rc*. Which version is the patch for?
Rick
Re: [PATCH 3/13] rework schedstats
Quoted 6 lines · Rick Lindsley
I have an updated userspace parser for this thing, if you are still keeping it on your website.
Sure, be happy to include it, thanks! Send it along. Is it for version 11 or version 12?
Version 12. I can send it to you next week. This was actually directed at Andrew, who was hosting my parser somewhere for a while. But it would be probably better if it just goes on your site.
Sorry, no update to your script because I can't write perl to save my life, let alone read it :|
Quoted 7 lines
Move balancing fields into struct
sched_domain, so we can get more useful results on systems with multiple domains (eg SMT+SMP, CMP+NUMA, SMP+NUMA, etc).It looks like you've also removed the stats for
pull_task()andwake_up_new_task(). Was this intentional, or accidental?
I didn't really think wunt_cnt or wunt_moved were very interesting with respect to the scheduler. wunt_cnt is more or less just the total number of forks, wunt_moved is a very rare situation where the new task is moved before being woken up.
The balance-on-fork schedstats code does add some more interesting stuff here.
pull_task() lost the "lost" counter because that is tricky to put into the context of sched-domains. It also isn't important in my current line of analysis because I'm just doing summations over all CPUs, so in that case your "gained" was always the same as "lost".
If you have uses for other counters, by all means send patches and we can discuss (not that I can imagine I'd have any objections).
Quoted 3 lines
I can't quite get the patch to apply cleanly against several variants of 2.6.10 or 2.6.11-rc*. Which version is the patch for?
It was 11-rc4, but it should fit on -rc5 OK too.
DHCP on multi homed host!
Hi,
Just had a question on multi homed host.
I want to have two network cards and two IPs. I want one IP to be
statically configured and the IP for card to be obtained by DHCP. Is this possible. Any references??
Regards,
Ravindra N.
Re: [PATCH 2/13] improve pinned task handling
In reply to Nick PigginQuoted 1 line · Nick Piggin
2/13
yeah, we need this. (Eventually someone should explore a different way to handle affine tasks as this is getting quirky, although it looks algorithmically impossible in an O(1) way.)
Ingo
Re: [PATCH 1/13] timestamp fixes
In reply to Nick PigginQuoted 2 lines · Nick Piggin
1/13
ugh, has this been tested? It needs the patch below.
Ingo
kernel/sched.c renamed from kernel/sched.c.orig +2 -2
@@ -2704,11 +2704,11 @@ need_resched_nonpreemptible:schedstat_inc (rq, sched_cnt); now =sched_clock (); -if (likely ((long long )now - prev->timestamp <NS_MAX_SLEEP_AVG )) +if (likely ((long long )now - prev->timestamp <NS_MAX_SLEEP_AVG )) { run_time = now - prev->timestamp;if (unlikely ((long long )now - prev->timestamp <0 )) run_time =0 ; -else + }else run_time =NS_MAX_SLEEP_AVG ;/*
Re: [PATCH 1/13] timestamp fixes
Quoted 7 lines · Ingo Molnar
Quoted 2 lines · Nick Piggin
1/13
ugh, has this been tested? It needs the patch below.
Yes. Which might also explain why I didn't see -ve intervals :( Thanks Ingo.
In the context of the whole patchset, testing has mainly been based around multiprocessor behaviour so this doesn't invalidate that.
Ingo Molnar
Acked-byRe: [PATCH 1/13] timestamp fixes
Quoted 6 lines · Nick Piggin
Quoted 5 lines · Ingo Molnar
Quoted 2 lines · Nick Piggin
1/13
Quoted 8 lines
Quoted 2 lines
ugh, has this been tested? It needs the patch below.
Yes. Which might also explain why I didn't see -ve intervals :( Thanks Ingo.
In the context of the whole patchset, testing has mainly been based around multiprocessor behaviour so this doesn't invalidate that.
nono, by 'this' i only meant that patch. The other ones look mainly OK, but obviously they need a _ton_ of testing.
these:
[PATCH 1/13] timestamp fixes
(+fix)
[PATCH 2/13] improve pinned task handling
[PATCH 3/13] rework schedstats
can go into BK right after 2.6.11 is released as they are fixes or norisk-improvements. [lets call them 'group A'] These three:
[PATCH 4/13] find_busiest_group fixlets
[PATCH 5/13] find_busiest_group cleanup
[PATCH 7/13] better active balancing heuristic
look pretty fine too and i'd suggest early BK integration too - but in theory they could impact things negatively so that's where immediate BK integration has to stop in the first phase, to get some feedback. [lets call them 'group B']
these:
[PATCH 6/13] no aggressive idle balancing
[PATCH 8/13] generalised CPU load averaging [PATCH 9/13] less affine wakups
[PATCH 10/13] remove aggressive idle balancing [PATCH 11/13] sched-domains aware balance-on-fork
[PATCH 12/13] schedstats additions for sched-balance-fork
[PATCH 13/13] basic tuning
change things radically, and i'm uneasy about them even in the 2.6.12 timeframe. [lets call them 'group C'] I'd suggest we give them a go in -mm and see how things go, so all of them get:
If things dont stabilize quickly then we need to do it piecemail wise. The only possible natural split seems to be to go for the running-task balancing changes first:
[PATCH 6/13] no aggressive idle balancing
[PATCH 8/13] generalised CPU load averaging
[PATCH 9/13] less affine wakups
[PATCH 10/13] remove aggressive idle balancing
[PATCH 13/13] basic tuning
perhaps #8 and relevant portions of #13 could be moved from group C into group B and thus hit BK early, but that would need remerging.
and then for the fork/clone-balancing changes:
[PATCH 11/13] sched-domains aware balance-on-fork [PATCH 12/13] schedstats additions for sched-balance-fork
a more finegrained splitup doesnt make much sense, as these groups are pretty compact conceptually.
But i expect fork/clone balancing to be almost certainly a problem. (We didnt get it right for all workloads in 2.6.7, and i think it cannot be gotten right currently either, without userspace API help - but i'd be happy to be proven wrong.)
(if you agree with my generic analysis then when you regenerate your patches next time please reorder them according to the flow above, and please try to insert future fixlets not end-of-stream but according to the conceptual grouping.)
Ingo