sched: wake-affine throttle
Subsystems (2)
PATCH 1/1 sched: wake-affine throttle
Log since RFC:
1. Throttle only when wake-affine failed. (thanks to PeterZ)
2. Do throttle inside wake_affine(). (thanks to PeterZ)
3. Other small fix.
Recently testing show that wake-affine stuff cause regression on pgbench, the hiding rat was finally catched out.
wake-affine stuff is always trying to pull wakee close to waker, by theory, this will benefit us if waker's cpu cached hot data for wakee, or the extreme ping-pong case.
However, the whole stuff is somewhat blindly, load balance is the only factor to be guaranteed, and since the stuff itself is time-consuming, some workload suffered, pgbench is just the one who has been found.
Thus, throttle the wake-affine stuff for such workload is necessary.
This patch introduced a new knob 'sysctl_sched_wake_affine_interval' with the default value 1ms (default minimum balance interval), which means wake-affine will keep silent for 1ms after it returned false.
By turning the new knob, those workload who suffered will have the chance to stop the regression.
Test:
Test with 12 cpu X86 server and tip 3.9.0-rc2. default base 1ms interval 10ms interval 100ms interval | db_size | clients | tps | | tps | | tps | | tps | +---------+---------+-------+- +-------+ +-------+ +-------+ | 21 MB | 1 | 10572 | | 10804 | | 10802 | | 10801 | | 21 MB | 2 | 21275 | | 21533 | | 21400 | | 21498 | | 21 MB | 4 | 41866 | | 42158 | | 42410 | | 42306 | | 21 MB | 8 | 53931 | | 55796 | | 58608 | +8.67% | 59916 | +11.10% | 21 MB | 12 | 50956 | | 52266 | | 54586 | +7.12% | 55982 | +9.86% | 21 MB | 16 | 49911 | | 52862 | +5.91% | 55668 | +11.53% | 57255 | +14.71% | 21 MB | 24 | 46046 | | 48820 | +6.02% | 54269 | +17.86% | 58113 | +26.21% | 21 MB | 32 | 43405 | | 46635 | +7.44% | 53690 | +23.70% | 57729 | +33.00% | 7483 MB | 1 | 7734 | | 8013 | | 8046 | | 7879 | | 7483 MB | 2 | 19375 | | 19459 | | 19448 | | 19421 | | 7483 MB | 4 | 37408 | | 37780 | | 37937 | | 37819 | | 7483 MB | 8 | 49033 | | 50389 | | 51636 | +5.31% | 52294 | +6.65% | 7483 MB | 12 | 45525 | | 47794 | +4.98% | 49828 | +9.45% | 50571 | +11.08% | 7483 MB | 16 | 45731 | | 47921 | +4.79% | 50203 | +9.78% | 52033 | +13.78% | 7483 MB | 24 | 41533 | | 44301 | +6.67% | 49697 | +19.66% | 53833 | +29.62% | 7483 MB | 32 | 36370 | | 38301 | +5.31% | 48146 | +32.38% | 52795 | +45.16% | 15 GB | 1 | 7576 | | 7926 | | 7722 | | 7969 | | 15 GB | 2 | 19157 | | 19284 | | 19294 | | 19304 | | 15 GB | 4 | 37285 | | 37539 | | 37281 | | 37508 | | 15 GB | 8 | 48718 | | 49176 | | 50836 | +4.35% | 51239 | +5.17% | 15 GB | 12 | 45167 | | 47180 | +4.45% | 49206 | +8.94% | 50126 | +10.98% | 15 GB | 16 | 45270 | | 47293 | +4.47% | 49638 | +9.65% | 51748 | +14.31% | 15 GB | 24 | 40984 | | 43366 | +5.81% | 49356 | +20.43% | 53157 | +29.70% | 15 GB | 32 | 35918 | | 37632 | +4.77% | 47923 | +33.42% | 52241 | +45.45%
--- include/linux/sched.h | 5 +++++ kernel/sched/fair.c | 31 +++++++++++++++++++++++++++++++ kernel/sysctl.c | 10 ++++++++++ 3 files changed, 46 insertions(+), 0 deletions(-)
include/linux/sched.h +5 -0
@@ -1197,6 +1197,10 @@ enum perf_event_task_context { perf_nr_task_contexts, }; +#ifdef CONFIG_SMP +extern unsigned int sysctl_sched_wake_affine_interval; +#endif +struct task_struct {volatile long state;/* -1 unrunnable, 0 runnable, >0 stopped */ void *stack;
@@ -1207,6 +1211,7 @@ struct task_struct {#ifdef CONFIG_SMP struct llist_node wake_entry;int on_cpu; +unsigned long next_wake_affine;#endif int on_rq;
kernel/sched/fair.c +31 -0
@@ -3087,6 +3087,22 @@ static inline unsigned long effective_load(struct task_group *tg, int cpu,#endif +/* +* Default is 1ms, to prevent the wake_affine() stuff working too frequently. +*/ +unsigned int sysctl_sched_wake_affine_interval =1U ; + +static inline int wake_affine_throttled (struct task_struct *p) +{ +return time_before (jiffies, p->next_wake_affine); +} + +static inline void wake_affine_throttle (struct task_struct *p) +{ + p->next_wake_affine = jiffies + +msecs_to_jiffies (sysctl_sched_wake_affine_interval); +} +static int wake_affine (struct sched_domain *sd,struct task_struct *p,int sync) {s64 this_load, load;
@@ -3096,6 +3112,9 @@ static int wake_affine(struct sched_domain *sd, struct task_struct *p, int sync)unsigned long weight;int balanced; +if (wake_affine_throttled (p)) +return 0 ; + idx = sd->wake_idx; this_cpu =smp_processor_id (); prev_cpu =task_cpu (p);
@@ -3167,6 +3186,18 @@ static int wake_affine(struct sched_domain *sd, struct task_struct *p, int sync)return 1 ; } + +/* +* wake_affine() stuff try to pull wakee to the cpu +* around waker, this will benefit us if the data +* cached on waker cpu is hot for wakee, or the extreme +* ping-pong case. +* +* However, do such blindly work too frequently will +* cause regression to some workload, thus, each time +* when wake_affine() failed, throttle it for a while. +*/ +wake_affine_throttle (p);return 0 ; }
kernel/sysctl.c +10 -0
@@ -437,6 +437,16 @@ static struct ctl_table kern_table[] = { .extra1 = &one, },#endif +#ifdef CONFIG_SMP + { + .procname = "sched_wake_affine_interval", + .data = &sysctl_sched_wake_affine_interval, + .maxlen = sizeof(unsigned int), + .mode = 0644, + .proc_handler = proc_dointvec_minmax, + .extra1 = &zero, + }, +#endif #ifdef CONFIG_PROVE_LOCKING { .procname = "prove_locking",
Signature · 1 line
1.7.4.1
Quoted 2 lines · Michael Wang
Suggested-by Peter Zijlstra <[email protected]>Signed-off-by Michael Wang <[email protected]>
BTW, could you try the kbulid, hackbench and aim for this?
Signature · 1 line
Thanks Alex
Quoted 5 lines · Alex Shi
Quoted 2 lines · Michael Wang
Suggested-by Peter Zijlstra <[email protected]>Signed-off-by Michael Wang <[email protected]>Reviewed-by Alex Shi <[email protected]>
Thanks for your review :)
Quoted 2 lines
BTW, could you try the kbulid, hackbench and aim for this?
Sure, the patch has already been tested with aim7, also the hackbench, kbench, and ebizzy, no notable changes on my box with the default 1ms interval.
Regards,
Michael Wang
Quoted 1 line
Quoted 4 lines · Michael Wang
Quoted 1 line
Quoted 1 line
BTW, could you try the kbulid, hackbench and aim for this?
Sure, the patch has already been tested with aim7, also the hackbench, kbench, and ebizzy, no notable changes on my box with the default 1ms interval.
That's fine.
Signature · 1 line
Thanks Alex
Peter Zijlstra
maintainerQuoted 2 lines · Michael Wang
| 15 GB | 32 | 35918 | | 37632 | +4.77% | 47923 | +33.42% | 52241 | +45.45%
So I don't get this... is wake_affine() once every milisecond _that_ expensive?
Seeing we get a 45%!! improvement out of once every 100ms that would mean we're like spending 1/3rd of our time in wake_affine()? that's preposterous. So what's happening?
Hi, Peter
Thanks for your reply :)
Quoted 10 lines · Peter Zijlstra
Quoted 2 lines · Michael Wang
| 15 GB | 32 | 35918 | | 37632 | +4.77% | 47923 | +33.42% | 52241 | +45.45%So I don't get this... is
wake_affine()once every milisecond _that_ expensive?Seeing we get a 45%!! improvement out of once every 100ms that would mean we're like spending 1/3rd of our time in
wake_affine()? that's preposterous. So what's happening?
Not all the regression was caused by overhead, adopt curr_cpu not prev_cpu for select_idle_sibling() is a more important reason for the regression of pgbench.
In other word, for pgbench, we waste time in wake_affine() and make the wrong decision at most of the time, the previously patch show wake_affine() do pull unrelated tasks together, that's good if current cpu still cached hot data for wakee, but that's not the case of the workload like pgbench.
The workload just don't satisfied the decision changed by wake-affine, the more wake-affine active, the more it suffered, that's why 100ms show better results than 1ms, but when reached some rate, the benefit and lost of wake-affine will be balanced.
Regards,
Michael Wang
Quoted 3 lines
Quoted 20 lines · Michael Wang
Hi, Peter
Thanks for your reply :)
Quoted 10 lines · Peter Zijlstra
Quoted 2 lines · Michael Wang
| 15 GB | 32 | 35918 | | 37632 | +4.77% | 47923 | +33.42% | 52241 | +45.45%So I don't get this... is
wake_affine()once every milisecond _that_ expensive?Seeing we get a 45%!! improvement out of once every 100ms that would mean we're like spending 1/3rd of our time in
wake_affine()? that's preposterous. So what's happening?Not all the regression was caused by overhead, adopt
curr_cpunotprev_cpuforselect_idle_sibling()is a more important reason for the regression of pgbench.
Quoted 5 lines
In other word, for pgbench, we waste time in
wake_affine()and make the wrong decision at most of the time, the previously patch showwake_affine()do pull unrelated tasks together, that's good if current cpu still cached hot data for wakee, but that's not the case of the workload like pgbench.
Please let me know if I failed to express my thought clearly.
I know it's hard to figure out why throttle could bring so many benefit, since the wake-affine stuff is a black box with too many unmeasurable factors, but that's actually the reason why we finally figure out this throttle idea, not the approach like wakeup-buddy, although both of them help to stop the regression.
It's fortunate that there is a benchmark could help to find out the regression, and now we have a simple and efficient approach ready for action ;-)
Regards,
Michael Wang
Quoted 13 lines
The workload just don't satisfied the decision changed by wake-affine, the more wake-affine active, the more it suffered, that's why 100ms show better results than 1ms, but when reached some rate, the benefit and lost of wake-affine will be balanced.
Regards,
Michael WangQuoted 3 lines
Quoted 27 lines · Michael Wang
Quoted 20 lines · Michael Wang
Hi, Peter
Thanks for your reply :)
Quoted 10 lines · Peter Zijlstra
Quoted 2 lines · Michael Wang
| 15 GB | 32 | 35918 | | 37632 | +4.77% | 47923 | +33.42% | 52241 | +45.45%So I don't get this... is
wake_affine()once every milisecond _that_ expensive?Seeing we get a 45%!! improvement out of once every 100ms that would mean we're like spending 1/3rd of our time in
wake_affine()? that's preposterous. So what's happening?Not all the regression was caused by overhead, adopt
curr_cpunotprev_cpuforselect_idle_sibling()is a more important reason for the regression of pgbench.Quoted 5 lines
In other word, for pgbench, we waste time in
wake_affine()and make the wrong decision at most of the time, the previously patch showwake_affine()do pull unrelated tasks together, that's good if current cpu still cached hot data for wakee, but that's not the case of the workload like pgbench.
Quoted 7 lines
Please let me know if I failed to express my thought clearly.
I know it's hard to figure out why throttle could bring so many benefit, since the wake-affine stuff is a black box with too many unmeasurable factors, but that's actually the reason why we finally figure out this throttle idea, not the approach like wakeup-buddy, although both of them help to stop the regression.
For that load, as soon as clients+server exceeds socket size, pull is doomed to always be a guaranteed loser. There simply is no way to win, some tasks must drag their data cross node no matter what you do, because there is one and only one source of data, so you can not possibly do anything but harm by pulling or in any other way disturbing task placement, because you will force tasks to re-heat their footprint every time you migrate someone with zero benefit to offset cost. That is why the closer you get to completely killing all migration, the better your throughput gets with this load.. you're killing the cost of migration in a situation there simply is no gain to be had.
That's why that wakeup-buddy thingy is a ~good idea. It will allow 1:1 buddies that can and do benefit from motion to pair up and jabber in a shared cache (though that motion needs slowing down too), _and_ detect the case where wakeup migration is utterly pointless. Just killing wakeup migration OTOH should (I'd say very emphatic will) hurt pgbench just as much, because spreading a smallish set which could share a cache across several nodes hurts things like pgbench via misses just as much as any other load.. it's just that once this load (or ilk) doesn't fit in a node, you're absolutely screwed as far as misses go, you will eat that because there simply is no other option.
Any migration is pointless for this thing once it exceeds socket size, and fairness plays a dominant role, is absolutely not throughputs best friend when any component of a load requires more CPU than the other components, which very definitely is the case with pgbench. Fairness hurts this thing a lot. That's why pgbench took a whopping huge hit when I fixed up select_idle_sibling() to not completely rape fast/light communicating tasks, it forced pgbench to face the consequences of a fair scheduler, by cutting off the escape routes that searching for _any_ even ever so briefly idle spot to place tasks such that wakeup preemption just didn't happen, and when we failed to pull, we instead did the very same thing on wakees original socket, thus providing pgbench the fairness escape mechanism that it needs.
When you wake to idle cores, you do not have a nanosecond resolution ultra fair scheduler, with the fairness price to be paid.. tasks run as long as they want to run, or at least full ticks, which of course makes the hard working load components a lot more productive. Hogs can be hogs. For pgbench run in 1:N mode, the hardest working load component is the mother of all work, the (singular) server. Any time 'mom' is not continuously working her little digital a$$ off to keep all those kids fed, you have a performance problem on your hands, the entire load stalls, lives and dies with one and only 'mom'.
-Mike
Hi, Mike
On 04/11/2013 03:30 PM, Mike Galbraith wrote:
[snip]
Quoted 20 lines
Quoted 8 lines
Please let me know if I failed to express my thought clearly.
I know it's hard to figure out why throttle could bring so many benefit, since the wake-affine stuff is a black box with too many unmeasurable factors, but that's actually the reason why we finally figure out this throttle idea, not the approach like wakeup-buddy, although both of them help to stop the regression.
For that load, as soon as clients+server exceeds socket size, pull is doomed to always be a guaranteed loser. There simply is no way to win, some tasks must drag their data cross node no matter what you do, because there is one and only one source of data, so you can not possibly do anything but harm by pulling or in any other way disturbing task placement, because you will force tasks to re-heat their footprint every time you migrate someone with zero benefit to offset cost. That is why the closer you get to completely killing all migration, the better your throughput gets with this load.. you're killing the cost of migration in a situation there simply is no gain to be had.
Quoted 10 lines
That's why that wakeup-buddy thingy is a ~good idea. It will allow 1:1 buddies that can and do benefit from motion to pair up and jabber in a shared cache (though that motion needs slowing down too), _and_ detect the case where wakeup migration is utterly pointless. Just killing wakeup migration OTOH should (I'd say very emphatic will) hurt pgbench just as much, because spreading a smallish set which could share a cache across several nodes hurts things like pgbench via misses just as much as any other load.. it's just that once this load (or ilk) doesn't fit in a node, you're absolutely screwed as far as misses go, you will eat that because there simply is no other option.
Wow, it's really hard to write down the logical behind, but you made it ;-)
The 1:N is a good reason to explain why the chance that wakee's hot data cached on curr_cpu is lower, and since it's just 'lower' not 'extinct', after the throttle interval large enough, it will be balanced, this could be proved, since during my test, when the interval become too big, the improvement start to drop.
Quoted 14 lines
Any migration is pointless for this thing once it exceeds socket size, and fairness plays a dominant role, is absolutely not throughputs best friend when any component of a load requires more CPU than the other components, which very definitely is the case with pgbench. Fairness hurts this thing a lot. That's why pgbench took a whopping huge hit when I fixed up
select_idle_sibling()to not completely rape fast/light communicating tasks, it forced pgbench to face the consequences of a fair scheduler, by cutting off the escape routes that searching for _any_ even ever so briefly idle spot to place tasks such that wakeup preemption just didn't happen, and when we failed to pull, we instead did the very same thing on wakees original socket, thus providing pgbench the fairness escape mechanism that it needs.
Quoted 9 lines
When you wake to idle cores, you do not have a nanosecond resolution ultra fair scheduler, with the fairness price to be paid.. tasks run as long as they want to run, or at least full ticks, which of course makes the hard working load components a lot more productive. Hogs can be hogs. For pgbench run in 1:N mode, the hardest working load component is the mother of all work, the (singular) server. Any time 'mom' is not continuously working her little digital a$$ off to keep all those kids fed, you have a performance problem on your hands, the entire load stalls, lives and dies with one and only 'mom'.
Hmm...that's an interesting point, the workload contain different 'priority' works, and depend on each other, if mother starving, all the kids could do nothing but wait for her, may be that's the reason why the benefit is so significant, since in such case, mother's little quicker respond will make all the kids happy :)
Regards,
Michael Wang
Quoted 3 lines
-Mike
Quoted 5 lines · Michael Wang
The 1:N is a good reason to explain why the chance that wakee's hot data cached on
curr_cpuis lower, and since it's just 'lower' not 'extinct', after the throttle interval large enough, it will be balanced, this could be proved, since during my test, when the interval become too big, the improvement start to drop.
Magnitude of improvement drops just because there's less damage done methinks. You'll eventually run out of measurable damage :)
Yes, it's not really extinct, you _can_ reap a gain, it's just not at all likely to work out. A more symetric load will fare better, but any 1:N thing just has to spread far and wide to have any chance to perform.
Quoted 5 lines
Hmm...that's an interesting point, the workload contain different 'priority' works, and depend on each other, if mother starving, all the kids could do nothing but wait for her, may be that's the reason why the benefit is so significant, since in such case, mother's little quicker respond will make all the kids happy :)
Exactly. The entire load is server latency bound. Keep the server on cpu, the load performs as best it can given unavoidable data miss cost.
-Mike
Quoted 14 lines · Mike Galbraith
Quoted 5 lines · Michael Wang
The 1:N is a good reason to explain why the chance that wakee's hot data cached on
curr_cpuis lower, and since it's just 'lower' not 'extinct', after the throttle interval large enough, it will be balanced, this could be proved, since during my test, when the interval become too big, the improvement start to drop.Magnitude of improvement drops just because there's less damage done methinks. You'll eventually run out of measurable damage :)
Yes, it's not really extinct, you _can_ reap a gain, it's just not at all likely to work out. A more symetric load will fare better, but any 1:N thing just has to spread far and wide to have any chance to perform.
Quoted 8 lines
Quoted 5 lines
Hmm...that's an interesting point, the workload contain different 'priority' works, and depend on each other, if mother starving, all the kids could do nothing but wait for her, may be that's the reason why the benefit is so significant, since in such case, mother's little quicker respond will make all the kids happy :)
Exactly. The entire load is server latency bound. Keep the server on cpu, the load performs as best it can given unavoidable data miss cost.
(ie serial producer, parallel consumer... choke point lies with utterly unscalable serial work producer)
Quoted 8 lines · Mike Galbraith
Quoted 5 lines · Michael Wang
The 1:N is a good reason to explain why the chance that wakee's hot data cached on
curr_cpuis lower, and since it's just 'lower' not 'extinct', after the throttle interval large enough, it will be balanced, this could be proved, since during my test, when the interval become too big, the improvement start to drop.
Quoted 6 lines
Magnitude of improvement drops just because there's less damage done methinks. You'll eventually run out of measurable damage :)
Yes, it's not really extinct, you _can_ reap a gain, it's just not at all likely to work out. A more symetric load will fare better, but any 1:N thing just has to spread far and wide to have any chance to perform.
Agree.
Quoted 8 lines
Quoted 5 lines
Hmm...that's an interesting point, the workload contain different 'priority' works, and depend on each other, if mother starving, all the kids could do nothing but wait for her, may be that's the reason why the benefit is so significant, since in such case, mother's little quicker respond will make all the kids happy :)
Exactly. The entire load is server latency bound. Keep the server on cpu, the load performs as best it can given unavoidable data miss cost.
Nice point :)
Regards,
Michael Wang
Quoted 3 lines
-Mike
Quoted 10 lines · Peter Zijlstra
Quoted 2 lines · Michael Wang
| 15 GB | 32 | 35918 | | 37632 | +4.77% | 47923 | +33.42% | 52241 | +45.45%So I don't get this... is
wake_affine()once every milisecond _that_ expensive?Seeing we get a 45%!! improvement out of once every 100ms that would mean we're like spending 1/3rd of our time in
wake_affine()? that's preposterous. So what's happening?
Hi, Peter
I think Mike has very well explained the reason why throttle bring us benefit and why the benefit looks so significant when interval get higher, could you please take a look at his mail and see whether it addressed this concern?
And thanks Mike again for the excellent analysis :)
Regards,
Michael Wang
Quoted 9 lines
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Quoted 16 lines · Michael Wang
Log since RFC:
1. Throttle only when wake-affine failed. (thanks to PeterZ)
2. Do throttle insidewake_affine(). (thanks to PeterZ)
3. Other small fix.Recently testing show that wake-affine stuff cause regression on pgbench, the hiding rat was finally catched out.
wake-affine stuff is always trying to pull wakee close to waker, by theory, this will benefit us if waker's cpu cached hot data for wakee, or the extreme ping-pong case.
However, the whole stuff is somewhat blindly, load balance is the only factor to be guaranteed, and since the stuff itself is time-consuming, some workload suffered, pgbench is just the one who has been found.
Quoted 8 lines
Thus, throttle the wake-affine stuff for such workload is necessary.
This patch introduced a new knob '
sysctl_sched_wake_affine_interval' with the default value 1ms (default minimum balance interval), which means wake-affine will keep silent for 1ms after it returned false.By turning the new knob, those workload who suffered will have the chance to stop the regression.
I have tested the latest tip 3.9.0-rc7, huge regression on pgbench is still there and this approach still works well, should we take the action now?
Regards,
Michael Wang
Quoted 142 lines
Test:
Test with 12 cpu X86 server and tip 3.9.0-rc2. default base 1ms interval 10ms interval 100ms interval | db_size | clients | tps | | tps | | tps | | tps | +---------+---------+-------+- +-------+ +-------+ +-------+ | 21 MB | 1 | 10572 | | 10804 | | 10802 | | 10801 | | 21 MB | 2 | 21275 | | 21533 | | 21400 | | 21498 | | 21 MB | 4 | 41866 | | 42158 | | 42410 | | 42306 | | 21 MB | 8 | 53931 | | 55796 | | 58608 | +8.67% | 59916 | +11.10% | 21 MB | 12 | 50956 | | 52266 | | 54586 | +7.12% | 55982 | +9.86% | 21 MB | 16 | 49911 | | 52862 | +5.91% | 55668 | +11.53% | 57255 | +14.71% | 21 MB | 24 | 46046 | | 48820 | +6.02% | 54269 | +17.86% | 58113 | +26.21% | 21 MB | 32 | 43405 | | 46635 | +7.44% | 53690 | +23.70% | 57729 | +33.00% | 7483 MB | 1 | 7734 | | 8013 | | 8046 | | 7879 | | 7483 MB | 2 | 19375 | | 19459 | | 19448 | | 19421 | | 7483 MB | 4 | 37408 | | 37780 | | 37937 | | 37819 | | 7483 MB | 8 | 49033 | | 50389 | | 51636 | +5.31% | 52294 | +6.65% | 7483 MB | 12 | 45525 | | 47794 | +4.98% | 49828 | +9.45% | 50571 | +11.08% | 7483 MB | 16 | 45731 | | 47921 | +4.79% | 50203 | +9.78% | 52033 | +13.78% | 7483 MB | 24 | 41533 | | 44301 | +6.67% | 49697 | +19.66% | 53833 | +29.62% | 7483 MB | 32 | 36370 | | 38301 | +5.31% | 48146 | +32.38% | 52795 | +45.16% | 15 GB | 1 | 7576 | | 7926 | | 7722 | | 7969 | | 15 GB | 2 | 19157 | | 19284 | | 19294 | | 19304 | | 15 GB | 4 | 37285 | | 37539 | | 37281 | | 37508 | | 15 GB | 8 | 48718 | | 49176 | | 50836 | +4.35% | 51239 | +5.17% | 15 GB | 12 | 45167 | | 47180 | +4.45% | 49206 | +8.94% | 50126 | +10.98% | 15 GB | 16 | 45270 | | 47293 | +4.47% | 49638 | +9.65% | 51748 | +14.31% | 15 GB | 24 | 40984 | | 43366 | +5.81% | 49356 | +20.43% | 53157 | +29.70% | 15 GB | 32 | 35918 | | 37632 | +4.77% | 47923 | +33.42% | 52241 | +45.45%Suggested-by Peter Zijlstra <[email protected]>Signed-off-by Michael Wang <[email protected]>--- include/linux/sched.h | 5 +++++ kernel/sched/fair.c | 31 +++++++++++++++++++++++++++++++ kernel/sysctl.c | 10 ++++++++++ 3 files changed, 46 insertions(+), 0 deletions(-)include/linux/sched.h +5
-0@@ -1197,6 +1197,10 @@ enum perf_event_task_context { perf_nr_task_contexts, }; +#ifdef CONFIG_SMP +extern unsigned int sysctl_sched_wake_affine_interval; +#endif +struct task_struct {volatile long state;/* -1 unrunnable, 0 runnable, >0 stopped */ void *stack;@@ -1207,6 +1211,7 @@ struct task_struct {#ifdef CONFIG_SMP struct llist_node wake_entry;int on_cpu; +unsigned long next_wake_affine;#endif int on_rq;kernel/sched/fair.c +31
-0@@ -3087,6 +3087,22 @@ static inline unsigned long effective_load(struct task_group *tg, int cpu,#endif +/* +* Default is 1ms, to prevent the wake_affine() stuff working too frequently. +*/ +unsigned int sysctl_sched_wake_affine_interval =1U ; + +static inline int wake_affine_throttled (struct task_struct *p) +{ +return time_before (jiffies, p->next_wake_affine); +} + +static inline void wake_affine_throttle (struct task_struct *p) +{ + p->next_wake_affine = jiffies + +msecs_to_jiffies (sysctl_sched_wake_affine_interval); +} +static int wake_affine (struct sched_domain *sd,struct task_struct *p,int sync) {s64 this_load, load;@@ -3096,6 +3112,9 @@ static int wake_affine(struct sched_domain *sd, struct task_struct *p, int sync)unsigned long weight;int balanced; +if (wake_affine_throttled (p)) +return 0 ; + idx = sd->wake_idx; this_cpu =smp_processor_id (); prev_cpu =task_cpu (p);@@ -3167,6 +3186,18 @@ static int wake_affine(struct sched_domain *sd, struct task_struct *p, int sync)return 1 ; } + +/* +* wake_affine() stuff try to pull wakee to the cpu +* around waker, this will benefit us if the data +* cached on waker cpu is hot for wakee, or the extreme +* ping-pong case. +* +* However, do such blindly work too frequently will +* cause regression to some workload, thus, each time +* when wake_affine() failed, throttle it for a while. +*/ +wake_affine_throttle (p);return 0 ; }kernel/sysctl.c +10
-0@@ -437,6 +437,16 @@ static struct ctl_table kern_table[] = { .extra1 = &one, },#endif +#ifdef CONFIG_SMP + { + .procname = "sched_wake_affine_interval", + .data = &sysctl_sched_wake_affine_interval, + .maxlen = sizeof(unsigned int), + .mode = 0644, + .proc_handler = proc_dointvec_minmax, + .extra1 = &zero, + }, +#endif #ifdef CONFIG_PROVE_LOCKING { .procname = "prove_locking",
Quoted 23 lines · Michael Wang
Quoted 22 lines · Michael Wang
Log since RFC:
1. Throttle only when wake-affine failed. (thanks to PeterZ)
2. Do throttle insidewake_affine(). (thanks to PeterZ)
3. Other small fix.Recently testing show that wake-affine stuff cause regression on pgbench, the hiding rat was finally catched out.
wake-affine stuff is always trying to pull wakee close to waker, by theory, this will benefit us if waker's cpu cached hot data for wakee, or the extreme ping-pong case.
However, the whole stuff is somewhat blindly, load balance is the only factor to be guaranteed, and since the stuff itself is time-consuming, some workload suffered, pgbench is just the one who has been found.
Thus, throttle the wake-affine stuff for such workload is necessary.
This patch introduced a new knob '
sysctl_sched_wake_affine_interval' with the default value 1ms (default minimum balance interval), which means wake-affine will keep silent for 1ms after it returned false.
Quoted 6 lines
Quoted 2 lines
By turning the new knob, those workload who suffered will have the chance to stop the regression.
I have tested the latest tip 3.9.0-rc7, huge regression on pgbench is still there and this approach still works well, should we take the action now?
(It's not a _regression_ per se, more of a long standing issue for this sort of load. Not that the load cares much what we call the problem;)
Quoted 147 lines
Regards,
Michael WangQuoted 142 lines
Test:
Test with 12 cpu X86 server and tip 3.9.0-rc2. default base 1ms interval 10ms interval 100ms interval | db_size | clients | tps | | tps | | tps | | tps | +---------+---------+-------+- +-------+ +-------+ +-------+ | 21 MB | 1 | 10572 | | 10804 | | 10802 | | 10801 | | 21 MB | 2 | 21275 | | 21533 | | 21400 | | 21498 | | 21 MB | 4 | 41866 | | 42158 | | 42410 | | 42306 | | 21 MB | 8 | 53931 | | 55796 | | 58608 | +8.67% | 59916 | +11.10% | 21 MB | 12 | 50956 | | 52266 | | 54586 | +7.12% | 55982 | +9.86% | 21 MB | 16 | 49911 | | 52862 | +5.91% | 55668 | +11.53% | 57255 | +14.71% | 21 MB | 24 | 46046 | | 48820 | +6.02% | 54269 | +17.86% | 58113 | +26.21% | 21 MB | 32 | 43405 | | 46635 | +7.44% | 53690 | +23.70% | 57729 | +33.00% | 7483 MB | 1 | 7734 | | 8013 | | 8046 | | 7879 | | 7483 MB | 2 | 19375 | | 19459 | | 19448 | | 19421 | | 7483 MB | 4 | 37408 | | 37780 | | 37937 | | 37819 | | 7483 MB | 8 | 49033 | | 50389 | | 51636 | +5.31% | 52294 | +6.65% | 7483 MB | 12 | 45525 | | 47794 | +4.98% | 49828 | +9.45% | 50571 | +11.08% | 7483 MB | 16 | 45731 | | 47921 | +4.79% | 50203 | +9.78% | 52033 | +13.78% | 7483 MB | 24 | 41533 | | 44301 | +6.67% | 49697 | +19.66% | 53833 | +29.62% | 7483 MB | 32 | 36370 | | 38301 | +5.31% | 48146 | +32.38% | 52795 | +45.16% | 15 GB | 1 | 7576 | | 7926 | | 7722 | | 7969 | | 15 GB | 2 | 19157 | | 19284 | | 19294 | | 19304 | | 15 GB | 4 | 37285 | | 37539 | | 37281 | | 37508 | | 15 GB | 8 | 48718 | | 49176 | | 50836 | +4.35% | 51239 | +5.17% | 15 GB | 12 | 45167 | | 47180 | +4.45% | 49206 | +8.94% | 50126 | +10.98% | 15 GB | 16 | 45270 | | 47293 | +4.47% | 49638 | +9.65% | 51748 | +14.31% | 15 GB | 24 | 40984 | | 43366 | +5.81% | 49356 | +20.43% | 53157 | +29.70% | 15 GB | 32 | 35918 | | 37632 | +4.77% | 47923 | +33.42% | 52241 | +45.45%Suggested-by Peter Zijlstra <[email protected]>Signed-off-by Michael Wang <[email protected]>--- include/linux/sched.h | 5 +++++ kernel/sched/fair.c | 31 +++++++++++++++++++++++++++++++ kernel/sysctl.c | 10 ++++++++++ 3 files changed, 46 insertions(+), 0 deletions(-)include/linux/sched.h +5
-0@@ -1197,6 +1197,10 @@ enum perf_event_task_context { perf_nr_task_contexts, }; +#ifdef CONFIG_SMP +extern unsigned int sysctl_sched_wake_affine_interval; +#endif +struct task_struct {volatile long state;/* -1 unrunnable, 0 runnable, >0 stopped */ void *stack;@@ -1207,6 +1211,7 @@ struct task_struct {#ifdef CONFIG_SMP struct llist_node wake_entry;int on_cpu; +unsigned long next_wake_affine;#endif int on_rq;kernel/sched/fair.c +31
-0@@ -3087,6 +3087,22 @@ static inline unsigned long effective_load(struct task_group *tg, int cpu,#endif +/* +* Default is 1ms, to prevent the wake_affine() stuff working too frequently. +*/ +unsigned int sysctl_sched_wake_affine_interval =1U ; + +static inline int wake_affine_throttled (struct task_struct *p) +{ +return time_before (jiffies, p->next_wake_affine); +} + +static inline void wake_affine_throttle (struct task_struct *p) +{ + p->next_wake_affine = jiffies + +msecs_to_jiffies (sysctl_sched_wake_affine_interval); +} +static int wake_affine (struct sched_domain *sd,struct task_struct *p,int sync) {s64 this_load, load;@@ -3096,6 +3112,9 @@ static int wake_affine(struct sched_domain *sd, struct task_struct *p, int sync)unsigned long weight;int balanced; +if (wake_affine_throttled (p)) +return 0 ; + idx = sd->wake_idx; this_cpu =smp_processor_id (); prev_cpu =task_cpu (p);@@ -3167,6 +3186,18 @@ static int wake_affine(struct sched_domain *sd, struct task_struct *p, int sync)return 1 ; } + +/* +* wake_affine() stuff try to pull wakee to the cpu +* around waker, this will benefit us if the data +* cached on waker cpu is hot for wakee, or the extreme +* ping-pong case. +* +* However, do such blindly work too frequently will +* cause regression to some workload, thus, each time +* when wake_affine() failed, throttle it for a while. +*/ +wake_affine_throttle (p);return 0 ; }kernel/sysctl.c +10
-0@@ -437,6 +437,16 @@ static struct ctl_table kern_table[] = { .extra1 = &one, },#endif +#ifdef CONFIG_SMP + { + .procname = "sched_wake_affine_interval", + .data = &sysctl_sched_wake_affine_interval, + .maxlen = sizeof(unsigned int), + .mode = 0644, + .proc_handler = proc_dointvec_minmax, + .extra1 = &zero, + }, +#endif #ifdef CONFIG_PROVE_LOCKING { .procname = "prove_locking",
Hi, Mike
Thanks for your reply :)
Quoted 27 lines · Mike Galbraith
Quoted 26 lines · Michael Wang
Quoted 24 lines · Michael Wang
Log since RFC:
1. Throttle only when wake-affine failed. (thanks to PeterZ)
2. Do throttle insidewake_affine(). (thanks to PeterZ)
3. Other small fix.Recently testing show that wake-affine stuff cause regression on pgbench, the hiding rat was finally catched out.
wake-affine stuff is always trying to pull wakee close to waker, by theory, this will benefit us if waker's cpu cached hot data for wakee, or the extreme ping-pong case.
However, the whole stuff is somewhat blindly, load balance is the only factor to be guaranteed, and since the stuff itself is time-consuming, some workload suffered, pgbench is just the one who has been found.
Thus, throttle the wake-affine stuff for such workload is necessary.
This patch introduced a new knob '
sysctl_sched_wake_affine_interval' with the default value 1ms (default minimum balance interval), which means wake-affine will keep silent for 1ms after it returned false.By turning the new knob, those workload who suffered will have the chance to stop the regression.
Quoted 6 lines
Quoted 3 lines
I have tested the latest tip 3.9.0-rc7, huge regression on pgbench is still there and this approach still works well, should we take the action now?
(It's not a _regression_ per se, more of a long standing issue for this sort of load. Not that the load cares much what we call the problem;)
Yeah, it's not general but for one kind of workload (or may be several), compared with the world without wake-affine, I will call it the regression caused by wake-affine on pgbench ;)
Regards,
Michael Wang
Quoted 156 lines
Quoted 139 lines
Regards,
Michael WangQuoted 134 lines
Test:
Test with 12 cpu X86 server and tip 3.9.0-rc2. default base 1ms interval 10ms interval 100ms interval | db_size | clients | tps | | tps | | tps | | tps | +---------+---------+-------+- +-------+ +-------+ +-------+ | 21 MB | 1 | 10572 | | 10804 | | 10802 | | 10801 | | 21 MB | 2 | 21275 | | 21533 | | 21400 | | 21498 | | 21 MB | 4 | 41866 | | 42158 | | 42410 | | 42306 | | 21 MB | 8 | 53931 | | 55796 | | 58608 | +8.67% | 59916 | +11.10% | 21 MB | 12 | 50956 | | 52266 | | 54586 | +7.12% | 55982 | +9.86% | 21 MB | 16 | 49911 | | 52862 | +5.91% | 55668 | +11.53% | 57255 | +14.71% | 21 MB | 24 | 46046 | | 48820 | +6.02% | 54269 | +17.86% | 58113 | +26.21% | 21 MB | 32 | 43405 | | 46635 | +7.44% | 53690 | +23.70% | 57729 | +33.00% | 7483 MB | 1 | 7734 | | 8013 | | 8046 | | 7879 | | 7483 MB | 2 | 19375 | | 19459 | | 19448 | | 19421 | | 7483 MB | 4 | 37408 | | 37780 | | 37937 | | 37819 | | 7483 MB | 8 | 49033 | | 50389 | | 51636 | +5.31% | 52294 | +6.65% | 7483 MB | 12 | 45525 | | 47794 | +4.98% | 49828 | +9.45% | 50571 | +11.08% | 7483 MB | 16 | 45731 | | 47921 | +4.79% | 50203 | +9.78% | 52033 | +13.78% | 7483 MB | 24 | 41533 | | 44301 | +6.67% | 49697 | +19.66% | 53833 | +29.62% | 7483 MB | 32 | 36370 | | 38301 | +5.31% | 48146 | +32.38% | 52795 | +45.16% | 15 GB | 1 | 7576 | | 7926 | | 7722 | | 7969 | | 15 GB | 2 | 19157 | | 19284 | | 19294 | | 19304 | | 15 GB | 4 | 37285 | | 37539 | | 37281 | | 37508 | | 15 GB | 8 | 48718 | | 49176 | | 50836 | +4.35% | 51239 | +5.17% | 15 GB | 12 | 45167 | | 47180 | +4.45% | 49206 | +8.94% | 50126 | +10.98% | 15 GB | 16 | 45270 | | 47293 | +4.47% | 49638 | +9.65% | 51748 | +14.31% | 15 GB | 24 | 40984 | | 43366 | +5.81% | 49356 | +20.43% | 53157 | +29.70% | 15 GB | 32 | 35918 | | 37632 | +4.77% | 47923 | +33.42% | 52241 | +45.45%Suggested-by Peter Zijlstra <[email protected]>Signed-off-by Michael Wang <[email protected]>--- include/linux/sched.h | 5 +++++ kernel/sched/fair.c | 31 +++++++++++++++++++++++++++++++ kernel/sysctl.c | 10 ++++++++++ 3 files changed, 46 insertions(+), 0 deletions(-)include/linux/sched.h +5
-0@@ -1197,6 +1197,10 @@ enum perf_event_task_context { perf_nr_task_contexts, }; +#ifdef CONFIG_SMP +extern unsigned int sysctl_sched_wake_affine_interval; +#endif +struct task_struct {volatile long state;/* -1 unrunnable, 0 runnable, >0 stopped */ void *stack;@@ -1207,6 +1211,7 @@ struct task_struct {#ifdef CONFIG_SMP struct llist_node wake_entry;int on_cpu; +unsigned long next_wake_affine;#endif int on_rq;kernel/sched/fair.c +31
-0@@ -3087,6 +3087,22 @@ static inline unsigned long effective_load(struct task_group *tg, int cpu,#endif +/* +* Default is 1ms, to prevent the wake_affine() stuff working too frequently. +*/ +unsigned int sysctl_sched_wake_affine_interval =1U ; + +static inline int wake_affine_throttled (struct task_struct *p) +{ +return time_before (jiffies, p->next_wake_affine); +} + +static inline void wake_affine_throttle (struct task_struct *p) +{ + p->next_wake_affine = jiffies + +msecs_to_jiffies (sysctl_sched_wake_affine_interval); +} +static int wake_affine (struct sched_domain *sd,struct task_struct *p,int sync) {s64 this_load, load;@@ -3096,6 +3112,9 @@ static int wake_affine(struct sched_domain *sd, struct task_struct *p, int sync)unsigned long weight;int balanced; +if (wake_affine_throttled (p)) +return 0 ; + idx = sd->wake_idx; this_cpu =smp_processor_id (); prev_cpu =task_cpu (p);@@ -3167,6 +3186,18 @@ static int wake_affine(struct sched_domain *sd, struct task_struct *p, int sync)return 1 ; } + +/* +* wake_affine() stuff try to pull wakee to the cpu +* around waker, this will benefit us if the data +* cached on waker cpu is hot for wakee, or the extreme +* ping-pong case. +* +* However, do such blindly work too frequently will +* cause regression to some workload, thus, each time +* when wake_affine() failed, throttle it for a while. +*/ +wake_affine_throttle (p);return 0 ; }kernel/sysctl.c +6
-0@@ -437,6 +437,16 @@ static struct ctl_table kern_table[] = { .extra1 = &one, },#endif +#ifdef CONFIG_SMP + { + .procname = "sched_wake_affine_interval", + .data = &sysctl_sched_wake_affine_interval, + .maxlen = sizeof(unsigned int), + .mode = 0644,Quoted 9 lines
kernel/sysctl.c static struct ctl_table kern_table[] = { · line 452
Quoted 8 lines
kernel/sysctl.c +4
-0+ .proc_handler = proc_dointvec_minmax, + .extra1 = &zero, + }, +#endif #ifdef CONFIG_PROVE_LOCKING { .procname = "prove_locking",--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Peter Zijlstra
maintainerOK,.. Ingo said that pipe-test was the original motivation for wake_affine() and since that's currently broken to pieces due to select_idle_sibling() is there still a benefit to having it at all?
Can anybody find any significant regression when simply killing wake_affine()?
Quoted 6 lines · Peter Zijlstra
OK,.. Ingo said that pipe-test was the original motivation for
wake_affine()and since that's currently broken to pieces due toselect_idle_sibling()is there still a benefit to having it at all?Can anybody find any significant regression when simply killing
wake_affine()?
I'd suggest doing a patch that does:
s/SD_WAKE_AFFINE/0*SD_WAKE_AFFINE
in all the relevant toplogy.h files, but otherwise keep the logic in place. That way it's easy to revert.
Thanks,
Ingo
Quoted 8 lines · Peter Zijlstra
OK,.. Ingo said that pipe-test was the original motivation for
wake_affine()and since that's currently broken to pieces due toselect_idle_sibling()is there still a benefit to having it at all?Can anybody find any significant regression when simply killing
wake_affine()?
This matches our experience; we've always tuned toward a more aggressive select_idle_sibling() for in-socket wake-ups.
We did recently turn it back on in the cross-socket case (e.g. SD_NODE) however as we were able to observe a benefit on a few workloads.
Quoted 7 lines · Peter Zijlstra
OK,.. Ingo said that pipe-test was the original motivation for
wake_affine()and since that's currently broken to pieces due toselect_idle_sibling()is there still a benefit to having it at all?Can anybody find any significant regression when simply killing
wake_affine()?
IMHO, reserve this stuff may be a better choice, it do bring benefit, pgbench is just some workload it also bring damage.
By throttle, we could reduce the damage meanwhile reserve the benefit, when reach some rate, they will be balanced and show best performance, I suppose just killing the stuff won't show that peak...
Besides, we still have many theory about the possible benefit of this stuff, along with some side-effect like speared behaviour, it has been there too long, the influence of killing it is unmeasurable (tend to be worse), I really prefer to reserve it...
And if we really want to kill it, we could just throttle it with an incredible interval, and make it unable to breathe any more ;-)
Regards,
Michael Wang
Quoted 8 lines
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Quoted 7 lines · Ingo Molnar
Quoted 4 lines · Peter Zijlstra
OK,.. Ingo said that pipe-test was the original motivation for
wake_affine()and since that's currently broken to pieces due toselect_idle_sibling()is there still a benefit to having it at all?
Quoted 6 lines
Quoted 2 lines
Can anybody find any significant regression when simply killing
wake_affine()?I'd suggest doing a patch that does:
s/SD_WAKE_AFFINE/0*SD_WAKE_AFFINE
But by doing this, we won't be able to find 'affine_sd' any more, that will also skip the select_idle_sibling() logical (in current code), isn't it?
If we really want to kill the stuff (I prefer not...), I suggest we forbidden the wake-affine by throttle it with an incredible interval, that's also easily to be reverted :)
Regards,
Michael Wang
Quoted 13 lines
in all the relevant
toplogy.hfiles, but otherwise keep the logic in place. That way it's easy to revert.Thanks,
Ingo
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Quoted 7 lines · Peter Zijlstra
OK,.. Ingo said that pipe-test was the original motivation for
wake_affine()and since that's currently broken to pieces due toselect_idle_sibling()is there still a benefit to having it at all?Can anybody find any significant regression when simply killing
wake_affine()?
Is it time for us to make the decision now?
Forgive me for urge the progress, but I really confused what is stopping us from adopt the throttle approach...
Is there any other option more simple, safe and faster?
Regards,
Michael Wang
Quoted 8 lines
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Quoted 7 lines · Peter Zijlstra
OK,.. Ingo said that pipe-test was the original motivation for
wake_affine()and since that's currently broken to pieces due toselect_idle_sibling()is there still a benefit to having it at all?Can anybody find any significant regression when simply killing
wake_affine()?
I got the proof that we could not simply killing the stuff (finally...).
It's the hackbench with a high pipe number, still on 12 cpu box, the result of "./hackbench 48 process 10000" is:
Running with 48*40 (== 1920) tasks. Time: 33.372
After killed the wake-affine, the result is:
Running with 48*40 (== 1920) tasks. Time: 38.205
About 14.48% performance dropped without wake-affine, I guess it was caused by the missing spread behaviour.
I've done the test for several times, also compared with the throttle approach, default 1ms interval still works very well, the regression on hackbench start to exceed 2% when interval become 100ms on my box, but please note the pgbench already gain a lot benefit at that time.
I think now we could say that wake-affine is useful, and we could not simply kill it.
So I still suggest we adopt the throttle approach, then we could make adjustment according to the demand.
And please let me know if there are any concerns ;-)
Regards,
Michael Wang
Quoted 8 lines
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Quoted 25 lines · Michael Wang
Quoted 7 lines · Peter Zijlstra
OK,.. Ingo said that pipe-test was the original motivation for
wake_affine()and since that's currently broken to pieces due toselect_idle_sibling()is there still a benefit to having it at all?Can anybody find any significant regression when simply killing
wake_affine()?I got the proof that we could not simply killing the stuff (finally...).
It's the hackbench with a high pipe number, still on 12 cpu box, the result of "./hackbench 48 process 10000" is:
Running with 48*40 (== 1920) tasks. Time: 33.372After killed the wake-affine, the result is:
Running with 48*40 (== 1920) tasks. Time: 38.205About 14.48% performance dropped without wake-affine, I guess it was caused by the missing spread behaviour.
Quoted 7 lines
I've done the test for several times, also compared with the throttle approach, default 1ms interval still works very well, the regression on hackbench start to exceed 2% when interval become 100ms on my box, but please note the pgbench already gain a lot benefit at that time.
I think now we could say that wake-affine is useful, and we could not simply kill it.
Oh, it's definitely useful. Communicating tasks deeply resent talking over interconnects (advanced tin cans and string). My little Q6600 box can even be described as dinky-numa given enough imagination.. place communicating tasks on different core2 "nodes" if you will, throughput falls through the floor. Shared L2 is quick like bunny, dram ain't.
-Mike
Quoted 6 lines · Mike Galbraith
Quoted 6 lines
I've done the test for several times, also compared with the throttle approach, default 1ms interval still works very well, the regression on hackbench start to exceed 2% when interval become 100ms on my box, but please note the pgbench already gain a lot benefit at that time.
Quoted 8 lines
Quoted 2 lines
I think now we could say that wake-affine is useful, and we could not simply kill it.
Oh, it's definitely useful. Communicating tasks deeply resent talking over interconnects (advanced tin cans and string). My little Q6600 box can even be described as dinky-numa given enough imagination.. place communicating tasks on different core2 "nodes" if you will, throughput falls through the floor. Shared L2 is quick like bunny, dram ain't.
Nice, so we got another proof to defend wake-affine now ;-)
Regards,
Michael Wang
Quoted 9 lines
-Mike
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
On 04/10/2013 11:30 AM, Michael Wang wrote:
Now long time has been passed since the first version, I'd like to do some summary about current states:
On a 12 cpu box with tip 3.9.0-rc7, test show that:
1. remove wake-affine stuff cause regression on hackbench (could be 15%).
2. reserve wake-affine stuff cause regression on pgbench (could be 45%).
And since neither remove or reserve is acceptable, this patch introduced a knob to throttle wake-affine stuff.
By turning the knob, we could benefit both hackbench and pgbench, or sacrifice one to significantly benefit another.
All similar workload which prefer or hate wake-affine behaviour could been cared.
If this approach caused any concerns, please let me know ;-)
Regards,
Michael Wang
Quoted 162 lines
Recently testing show that wake-affine stuff cause regression on pgbench, the hiding rat was finally catched out.
wake-affine stuff is always trying to pull wakee close to waker, by theory, this will benefit us if waker's cpu cached hot data for wakee, or the extreme ping-pong case.
However, the whole stuff is somewhat blindly, load balance is the only factor to be guaranteed, and since the stuff itself is time-consuming, some workload suffered, pgbench is just the one who has been found.
Thus, throttle the wake-affine stuff for such workload is necessary.
This patch introduced a new knob '
sysctl_sched_wake_affine_interval' with the default value 1ms (default minimum balance interval), which means wake-affine will keep silent for 1ms after it returned false.By turning the new knob, those workload who suffered will have the chance to stop the regression.
Test:
Test with 12 cpu X86 server and tip 3.9.0-rc2. default base 1ms interval 10ms interval 100ms interval | db_size | clients | tps | | tps | | tps | | tps | +---------+---------+-------+- +-------+ +-------+ +-------+ | 21 MB | 1 | 10572 | | 10804 | | 10802 | | 10801 | | 21 MB | 2 | 21275 | | 21533 | | 21400 | | 21498 | | 21 MB | 4 | 41866 | | 42158 | | 42410 | | 42306 | | 21 MB | 8 | 53931 | | 55796 | | 58608 | +8.67% | 59916 | +11.10% | 21 MB | 12 | 50956 | | 52266 | | 54586 | +7.12% | 55982 | +9.86% | 21 MB | 16 | 49911 | | 52862 | +5.91% | 55668 | +11.53% | 57255 | +14.71% | 21 MB | 24 | 46046 | | 48820 | +6.02% | 54269 | +17.86% | 58113 | +26.21% | 21 MB | 32 | 43405 | | 46635 | +7.44% | 53690 | +23.70% | 57729 | +33.00% | 7483 MB | 1 | 7734 | | 8013 | | 8046 | | 7879 | | 7483 MB | 2 | 19375 | | 19459 | | 19448 | | 19421 | | 7483 MB | 4 | 37408 | | 37780 | | 37937 | | 37819 | | 7483 MB | 8 | 49033 | | 50389 | | 51636 | +5.31% | 52294 | +6.65% | 7483 MB | 12 | 45525 | | 47794 | +4.98% | 49828 | +9.45% | 50571 | +11.08% | 7483 MB | 16 | 45731 | | 47921 | +4.79% | 50203 | +9.78% | 52033 | +13.78% | 7483 MB | 24 | 41533 | | 44301 | +6.67% | 49697 | +19.66% | 53833 | +29.62% | 7483 MB | 32 | 36370 | | 38301 | +5.31% | 48146 | +32.38% | 52795 | +45.16% | 15 GB | 1 | 7576 | | 7926 | | 7722 | | 7969 | | 15 GB | 2 | 19157 | | 19284 | | 19294 | | 19304 | | 15 GB | 4 | 37285 | | 37539 | | 37281 | | 37508 | | 15 GB | 8 | 48718 | | 49176 | | 50836 | +4.35% | 51239 | +5.17% | 15 GB | 12 | 45167 | | 47180 | +4.45% | 49206 | +8.94% | 50126 | +10.98% | 15 GB | 16 | 45270 | | 47293 | +4.47% | 49638 | +9.65% | 51748 | +14.31% | 15 GB | 24 | 40984 | | 43366 | +5.81% | 49356 | +20.43% | 53157 | +29.70% | 15 GB | 32 | 35918 | | 37632 | +4.77% | 47923 | +33.42% | 52241 | +45.45%Suggested-by Peter Zijlstra <[email protected]>Signed-off-by Michael Wang <[email protected]>--- include/linux/sched.h | 5 +++++ kernel/sched/fair.c | 31 +++++++++++++++++++++++++++++++ kernel/sysctl.c | 10 ++++++++++ 3 files changed, 46 insertions(+), 0 deletions(-)include/linux/sched.h +5
-0@@ -1197,6 +1197,10 @@ enum perf_event_task_context { perf_nr_task_contexts, }; +#ifdef CONFIG_SMP +extern unsigned int sysctl_sched_wake_affine_interval; +#endif +struct task_struct {volatile long state;/* -1 unrunnable, 0 runnable, >0 stopped */ void *stack;@@ -1207,6 +1211,7 @@ struct task_struct {#ifdef CONFIG_SMP struct llist_node wake_entry;int on_cpu; +unsigned long next_wake_affine;#endif int on_rq;kernel/sched/fair.c +31
-0@@ -3087,6 +3087,22 @@ static inline unsigned long effective_load(struct task_group *tg, int cpu,#endif +/* +* Default is 1ms, to prevent the wake_affine() stuff working too frequently. +*/ +unsigned int sysctl_sched_wake_affine_interval =1U ; + +static inline int wake_affine_throttled (struct task_struct *p) +{ +return time_before (jiffies, p->next_wake_affine); +} + +static inline void wake_affine_throttle (struct task_struct *p) +{ + p->next_wake_affine = jiffies + +msecs_to_jiffies (sysctl_sched_wake_affine_interval); +} +static int wake_affine (struct sched_domain *sd,struct task_struct *p,int sync) {s64 this_load, load;@@ -3096,6 +3112,9 @@ static int wake_affine(struct sched_domain *sd, struct task_struct *p, int sync)unsigned long weight;int balanced; +if (wake_affine_throttled (p)) +return 0 ; + idx = sd->wake_idx; this_cpu =smp_processor_id (); prev_cpu =task_cpu (p);@@ -3167,6 +3186,18 @@ static int wake_affine(struct sched_domain *sd, struct task_struct *p, int sync)return 1 ; } + +/* +* wake_affine() stuff try to pull wakee to the cpu +* around waker, this will benefit us if the data +* cached on waker cpu is hot for wakee, or the extreme +* ping-pong case. +* +* However, do such blindly work too frequently will +* cause regression to some workload, thus, each time +* when wake_affine() failed, throttle it for a while. +*/ +wake_affine_throttle (p);return 0 ; }kernel/sysctl.c +10
-0@@ -437,6 +437,16 @@ static struct ctl_table kern_table[] = { .extra1 = &one, },#endif +#ifdef CONFIG_SMP + { + .procname = "sched_wake_affine_interval", + .data = &sysctl_sched_wake_affine_interval, + .maxlen = sizeof(unsigned int), + .mode = 0644, + .proc_handler = proc_dointvec_minmax, + .extra1 = &zero, + }, +#endif #ifdef CONFIG_PROVE_LOCKING { .procname = "prove_locking",
Quoted 13 lines · Michael Wang
On 04/10/2013 11:30 AM, Michael Wang wrote:
Now long time has been passed since the first version, I'd like to do some summary about current states:
On a 12 cpu box with tip 3.9.0-rc7, test show that:
1. remove wake-affine stuff cause regression on hackbench (could be 15%).
2. reserve wake-affine stuff cause regression on pgbench (could be 45%).And since neither remove or reserve is acceptable, this patch introduced a knob to throttle wake-affine stuff.
Quoted 7 lines
By turning the knob, we could benefit both hackbench and pgbench, or sacrifice one to significantly benefit another.
All similar workload which prefer or hate wake-affine behaviour could been cared.
If this approach caused any concerns, please let me know ;-)
I wonder if throttling on failure is the way to go. Note the minimal gain for pgbench with the default 1ms throttle interval. It's not very effective out of the box for the load type it's targeted to help, and people generally don't twiddle scheduler knobs. If you throttle on success, you directly restrict migration frequency without that being affected by what other tasks are doing. Seems that would be a bit more effective.
(I still like the wakeup buddy thing, it's more effective because it adds and uses knowledge, though without the knob, cache domain size. Peter is right about the interrupt wakeups though, that could very easily cause regressions, dirt simple throttle is much safer).
-Mike
Hi, Mike
Thanks for your reply.
On 05/03/2013 01:01 PM, Mike Galbraith wrote:
[snip]
Quoted 10 lines
Quoted 2 lines
If this approach caused any concerns, please let me know ;-)
I wonder if throttling on failure is the way to go. Note the minimal gain for pgbench with the default 1ms throttle interval. It's not very effective out of the box for the load type it's targeted to help, and people generally don't twiddle scheduler knobs. If you throttle on success, you directly restrict migration frequency without that being affected by what other tasks are doing. Seems that would be a bit more effective.
This is a good timing to make some conclusion for this problem ;-)
Let's suppose when wake-affine failed, next time slice got a higher failure chance, then whether throttle on failure could be the question like:
throttle interval should cover more failure timing or more success timing?
Obviously we should cover more failure timing, since it's just wasting cycle and change nothing.
However, I used to concern about the damage of succeed wake-affine at that rapid, sure it also contain the benefit, but which one is bigger?
Now if we look at the RFC version which throttle on succeed, for pgbench, we could find that the default 1ms benefit is < 5%, while the current version which throttle on failure bring 7% at most.
And that eliminate my concern :)
Quoted 5 lines
(I still like the wakeup buddy thing, it's more effective because it adds and uses knowledge, though without the knob, cache domain size. Peter is right about the interrupt wakeups though, that could very easily cause regressions, dirt simple throttle is much safer).
Exactly, dark issue deserve dark solution, let darkness guide him...
Regards,
Michael Wang
Quoted 3 lines
-Mike
Quoted 28 lines · Michael Wang
Hi, Mike
Thanks for your reply.
On 05/03/2013 01:01 PM, Mike Galbraith wrote:
[snip]Quoted 10 lines
Quoted 2 lines
If this approach caused any concerns, please let me know ;-)
I wonder if throttling on failure is the way to go. Note the minimal gain for pgbench with the default 1ms throttle interval. It's not very effective out of the box for the load type it's targeted to help, and people generally don't twiddle scheduler knobs. If you throttle on success, you directly restrict migration frequency without that being affected by what other tasks are doing. Seems that would be a bit more effective.
This is a good timing to make some conclusion for this problem ;-)
Let's suppose when wake-affine failed, next time slice got a higher failure chance, then whether throttle on failure could be the question like:
throttle interval should cover more failure timing or more success timing?
Obviously we should cover more failure timing, since it's just wasting cycle and change nothing.
Quoted 6 lines
However, I used to concern about the damage of succeed wake-affine at that rapid, sure it also contain the benefit, but which one is bigger?
Now if we look at the RFC version which throttle on succeed, for pgbench, we could find that the default 1ms benefit is < 5%, while the current version which throttle on failure bring 7% at most.
OK, so scratch that thought. Would still be good to find a dirt simple dirt cheap way to increase effectiveness a bit, and eliminate the knob. Until a better idea comes along, this helps pgbench some, and will also help fast movers ala tbench on AMD, where select_idle_sibling() wasn't particularly wonderful per my measurements.
-Mike
Quoted 36 lines · Mike Galbraith
Quoted 28 lines · Michael Wang
Hi, Mike
Thanks for your reply.
On 05/03/2013 01:01 PM, Mike Galbraith wrote:
[snip]Quoted 10 lines
Quoted 2 lines
If this approach caused any concerns, please let me know ;-)
I wonder if throttling on failure is the way to go. Note the minimal gain for pgbench with the default 1ms throttle interval. It's not very effective out of the box for the load type it's targeted to help, and people generally don't twiddle scheduler knobs. If you throttle on success, you directly restrict migration frequency without that being affected by what other tasks are doing. Seems that would be a bit more effective.
This is a good timing to make some conclusion for this problem ;-)
Let's suppose when wake-affine failed, next time slice got a higher failure chance, then whether throttle on failure could be the question like:
throttle interval should cover more failure timing or more success timing?
Obviously we should cover more failure timing, since it's just wasting cycle and change nothing.
Quoted 6 lines
However, I used to concern about the damage of succeed wake-affine at that rapid, sure it also contain the benefit, but which one is bigger?
Now if we look at the RFC version which throttle on succeed, for pgbench, we could find that the default 1ms benefit is < 5%, while the current version which throttle on failure bring 7% at most.
Quoted 5 lines
OK, so scratch that thought. Would still be good to find a dirt simple dirt cheap way to increase effectiveness a bit, and eliminate the knob. Until a better idea comes along, this helps pgbench some, and will also help fast movers ala tbench on AMD, where
select_idle_sibling()wasn't particularly wonderful per my measurements.
Yep, another advantage of this approach is simple, when later we figure out the better idea, it could be easily replaced, for now, I would prefer to use it as an urgent rescue for the 'suffered workload' ;-)
Regards,
Michael Wang
Quoted 9 lines
-Mike
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Quoted 13 lines · Michael Wang
On 04/10/2013 11:30 AM, Michael Wang wrote:
Now long time has been passed since the first version, I'd like to do some summary about current states:
On a 12 cpu box with tip 3.9.0-rc7, test show that:
1. remove wake-affine stuff cause regression on hackbench (could be 15%).
2. reserve wake-affine stuff cause regression on pgbench (could be 45%).And since neither remove or reserve is acceptable, this patch introduced a knob to throttle wake-affine stuff.
Quoted 7 lines
By turning the knob, we could benefit both hackbench and pgbench, or sacrifice one to significantly benefit another.
All similar workload which prefer or hate wake-affine behaviour could been cared.
If this approach caused any concerns, please let me know ;-)
Is there any more concerns? or should we take some action now?
Regards,
Michael Wang
Quoted 173 lines
Regards,
Michael WangQuoted 152 lines
Recently testing show that wake-affine stuff cause regression on pgbench, the hiding rat was finally catched out.
wake-affine stuff is always trying to pull wakee close to waker, by theory, this will benefit us if waker's cpu cached hot data for wakee, or the extreme ping-pong case.
However, the whole stuff is somewhat blindly, load balance is the only factor to be guaranteed, and since the stuff itself is time-consuming, some workload suffered, pgbench is just the one who has been found.
Thus, throttle the wake-affine stuff for such workload is necessary.
This patch introduced a new knob '
sysctl_sched_wake_affine_interval' with the default value 1ms (default minimum balance interval), which means wake-affine will keep silent for 1ms after it returned false.By turning the new knob, those workload who suffered will have the chance to stop the regression.
Test:
Test with 12 cpu X86 server and tip 3.9.0-rc2. default base 1ms interval 10ms interval 100ms interval | db_size | clients | tps | | tps | | tps | | tps | +---------+---------+-------+- +-------+ +-------+ +-------+ | 21 MB | 1 | 10572 | | 10804 | | 10802 | | 10801 | | 21 MB | 2 | 21275 | | 21533 | | 21400 | | 21498 | | 21 MB | 4 | 41866 | | 42158 | | 42410 | | 42306 | | 21 MB | 8 | 53931 | | 55796 | | 58608 | +8.67% | 59916 | +11.10% | 21 MB | 12 | 50956 | | 52266 | | 54586 | +7.12% | 55982 | +9.86% | 21 MB | 16 | 49911 | | 52862 | +5.91% | 55668 | +11.53% | 57255 | +14.71% | 21 MB | 24 | 46046 | | 48820 | +6.02% | 54269 | +17.86% | 58113 | +26.21% | 21 MB | 32 | 43405 | | 46635 | +7.44% | 53690 | +23.70% | 57729 | +33.00% | 7483 MB | 1 | 7734 | | 8013 | | 8046 | | 7879 | | 7483 MB | 2 | 19375 | | 19459 | | 19448 | | 19421 | | 7483 MB | 4 | 37408 | | 37780 | | 37937 | | 37819 | | 7483 MB | 8 | 49033 | | 50389 | | 51636 | +5.31% | 52294 | +6.65% | 7483 MB | 12 | 45525 | | 47794 | +4.98% | 49828 | +9.45% | 50571 | +11.08% | 7483 MB | 16 | 45731 | | 47921 | +4.79% | 50203 | +9.78% | 52033 | +13.78% | 7483 MB | 24 | 41533 | | 44301 | +6.67% | 49697 | +19.66% | 53833 | +29.62% | 7483 MB | 32 | 36370 | | 38301 | +5.31% | 48146 | +32.38% | 52795 | +45.16% | 15 GB | 1 | 7576 | | 7926 | | 7722 | | 7969 | | 15 GB | 2 | 19157 | | 19284 | | 19294 | | 19304 | | 15 GB | 4 | 37285 | | 37539 | | 37281 | | 37508 | | 15 GB | 8 | 48718 | | 49176 | | 50836 | +4.35% | 51239 | +5.17% | 15 GB | 12 | 45167 | | 47180 | +4.45% | 49206 | +8.94% | 50126 | +10.98% | 15 GB | 16 | 45270 | | 47293 | +4.47% | 49638 | +9.65% | 51748 | +14.31% | 15 GB | 24 | 40984 | | 43366 | +5.81% | 49356 | +20.43% | 53157 | +29.70% | 15 GB | 32 | 35918 | | 37632 | +4.77% | 47923 | +33.42% | 52241 | +45.45%Suggested-by Peter Zijlstra <[email protected]>Signed-off-by Michael Wang <[email protected]>--- include/linux/sched.h | 5 +++++ kernel/sched/fair.c | 31 +++++++++++++++++++++++++++++++ kernel/sysctl.c | 10 ++++++++++ 3 files changed, 46 insertions(+), 0 deletions(-)include/linux/sched.h +5
-0@@ -1197,6 +1197,10 @@ enum perf_event_task_context { perf_nr_task_contexts, }; +#ifdef CONFIG_SMP +extern unsigned int sysctl_sched_wake_affine_interval; +#endif +struct task_struct {volatile long state;/* -1 unrunnable, 0 runnable, >0 stopped */ void *stack;@@ -1207,6 +1211,7 @@ struct task_struct {#ifdef CONFIG_SMP struct llist_node wake_entry;int on_cpu; +unsigned long next_wake_affine;#endif int on_rq;kernel/sched/fair.c +31
-0@@ -3087,6 +3087,22 @@ static inline unsigned long effective_load(struct task_group *tg, int cpu,#endif +/* +* Default is 1ms, to prevent the wake_affine() stuff working too frequently. +*/ +unsigned int sysctl_sched_wake_affine_interval =1U ; + +static inline int wake_affine_throttled (struct task_struct *p) +{ +return time_before (jiffies, p->next_wake_affine); +} + +static inline void wake_affine_throttle (struct task_struct *p) +{ + p->next_wake_affine = jiffies + +msecs_to_jiffies (sysctl_sched_wake_affine_interval); +} +static int wake_affine (struct sched_domain *sd,struct task_struct *p,int sync) {s64 this_load, load;@@ -3096,6 +3112,9 @@ static int wake_affine(struct sched_domain *sd, struct task_struct *p, int sync)unsigned long weight;int balanced; +if (wake_affine_throttled (p)) +return 0 ; + idx = sd->wake_idx; this_cpu =smp_processor_id (); prev_cpu =task_cpu (p);@@ -3167,6 +3186,18 @@ static int wake_affine(struct sched_domain *sd, struct task_struct *p, int sync)return 1 ; } + +/* +* wake_affine() stuff try to pull wakee to the cpu +* around waker, this will benefit us if the data +* cached on waker cpu is hot for wakee, or the extreme +* ping-pong case. +* +* However, do such blindly work too frequently will +* cause regression to some workload, thus, each time +* when wake_affine() failed, throttle it for a while. +*/ +wake_affine_throttle (p);return 0 ; }kernel/sysctl.c +3
-0@@ -437,6 +437,16 @@ static struct ctl_table kern_table[] = { .extra1 = &one, },#endif +#ifdef CONFIG_SMP + { + .procname = "sched_wake_affine_interval",kernel/sysctl.c static struct ctl_table kern_table[] = { · line 452
Quoted 10 lines
kernel/sysctl.c +7
-0+ .data = &sysctl_sched_wake_affine_interval, + .maxlen = sizeof(unsigned int), + .mode = 0644, + .proc_handler = proc_dointvec_minmax, + .extra1 = &zero, + }, +#endif #ifdef CONFIG_PROVE_LOCKING { .procname = "prove_locking",--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Quoted 17 lines · Michael Wang
Quoted 16 lines · Michael Wang
On 04/10/2013 11:30 AM, Michael Wang wrote:
Now long time has been passed since the first version, I'd like to do some summary about current states:
On a 12 cpu box with tip 3.9.0-rc7, test show that:
1. remove wake-affine stuff cause regression on hackbench (could be 15%).
2. reserve wake-affine stuff cause regression on pgbench (could be 45%).And since neither remove or reserve is acceptable, this patch introduced a knob to throttle wake-affine stuff.
By turning the knob, we could benefit both hackbench and pgbench, or sacrifice one to significantly benefit another.
Quoted 6 lines
Quoted 4 lines
All similar workload which prefer or hate wake-affine behaviour could been cared.
If this approach caused any concerns, please let me know ;-)
Is there any more concerns? or should we take some action now?
Seems like folks has no more concerns :)
Peter, would you like to pick the patch? or please give some comments may be?
Regards,
Michael Wang
Quoted 184 lines
Regards,
Michael WangQuoted 164 lines
Regards,
Michael WangQuoted 160 lines
Recently testing show that wake-affine stuff cause regression on pgbench, the hiding rat was finally catched out.
wake-affine stuff is always trying to pull wakee close to waker, by theory, this will benefit us if waker's cpu cached hot data for wakee, or the extreme ping-pong case.
However, the whole stuff is somewhat blindly, load balance is the only factor to be guaranteed, and since the stuff itself is time-consuming, some workload suffered, pgbench is just the one who has been found.
Thus, throttle the wake-affine stuff for such workload is necessary.
This patch introduced a new knob '
sysctl_sched_wake_affine_interval' with the default value 1ms (default minimum balance interval), which means wake-affine will keep silent for 1ms after it returned false.By turning the new knob, those workload who suffered will have the chance to stop the regression.
Test:
Test with 12 cpu X86 server and tip 3.9.0-rc2. default base 1ms interval 10ms interval 100ms interval | db_size | clients | tps | | tps | | tps | | tps | +---------+---------+-------+- +-------+ +-------+ +-------+ | 21 MB | 1 | 10572 | | 10804 | | 10802 | | 10801 | | 21 MB | 2 | 21275 | | 21533 | | 21400 | | 21498 | | 21 MB | 4 | 41866 | | 42158 | | 42410 | | 42306 | | 21 MB | 8 | 53931 | | 55796 | | 58608 | +8.67% | 59916 | +11.10% | 21 MB | 12 | 50956 | | 52266 | | 54586 | +7.12% | 55982 | +9.86% | 21 MB | 16 | 49911 | | 52862 | +5.91% | 55668 | +11.53% | 57255 | +14.71% | 21 MB | 24 | 46046 | | 48820 | +6.02% | 54269 | +17.86% | 58113 | +26.21% | 21 MB | 32 | 43405 | | 46635 | +7.44% | 53690 | +23.70% | 57729 | +33.00% | 7483 MB | 1 | 7734 | | 8013 | | 8046 | | 7879 | | 7483 MB | 2 | 19375 | | 19459 | | 19448 | | 19421 | | 7483 MB | 4 | 37408 | | 37780 | | 37937 | | 37819 | | 7483 MB | 8 | 49033 | | 50389 | | 51636 | +5.31% | 52294 | +6.65% | 7483 MB | 12 | 45525 | | 47794 | +4.98% | 49828 | +9.45% | 50571 | +11.08% | 7483 MB | 16 | 45731 | | 47921 | +4.79% | 50203 | +9.78% | 52033 | +13.78% | 7483 MB | 24 | 41533 | | 44301 | +6.67% | 49697 | +19.66% | 53833 | +29.62% | 7483 MB | 32 | 36370 | | 38301 | +5.31% | 48146 | +32.38% | 52795 | +45.16% | 15 GB | 1 | 7576 | | 7926 | | 7722 | | 7969 | | 15 GB | 2 | 19157 | | 19284 | | 19294 | | 19304 | | 15 GB | 4 | 37285 | | 37539 | | 37281 | | 37508 | | 15 GB | 8 | 48718 | | 49176 | | 50836 | +4.35% | 51239 | +5.17% | 15 GB | 12 | 45167 | | 47180 | +4.45% | 49206 | +8.94% | 50126 | +10.98% | 15 GB | 16 | 45270 | | 47293 | +4.47% | 49638 | +9.65% | 51748 | +14.31% | 15 GB | 24 | 40984 | | 43366 | +5.81% | 49356 | +20.43% | 53157 | +29.70% | 15 GB | 32 | 35918 | | 37632 | +4.77% | 47923 | +33.42% | 52241 | +45.45%Suggested-by Peter Zijlstra <[email protected]>Signed-off-by Michael Wang <[email protected]>--- include/linux/sched.h | 5 +++++ kernel/sched/fair.c | 31 +++++++++++++++++++++++++++++++ kernel/sysctl.c | 10 ++++++++++ 3 files changed, 46 insertions(+), 0 deletions(-)include/linux/sched.h +5
-0@@ -1197,6 +1197,10 @@ enum perf_event_task_context { perf_nr_task_contexts, }; +#ifdef CONFIG_SMP +extern unsigned int sysctl_sched_wake_affine_interval; +#endif +struct task_struct {volatile long state;/* -1 unrunnable, 0 runnable, >0 stopped */ void *stack;@@ -1207,6 +1211,7 @@ struct task_struct {#ifdef CONFIG_SMP struct llist_node wake_entry;int on_cpu; +unsigned long next_wake_affine;#endif int on_rq;kernel/sched/fair.c +31
-0@@ -3087,6 +3087,22 @@ static inline unsigned long effective_load(struct task_group *tg, int cpu,#endif +/* +* Default is 1ms, to prevent the wake_affine() stuff working too frequently. +*/ +unsigned int sysctl_sched_wake_affine_interval =1U ; + +static inline int wake_affine_throttled (struct task_struct *p) +{ +return time_before (jiffies, p->next_wake_affine); +} + +static inline void wake_affine_throttle (struct task_struct *p) +{ + p->next_wake_affine = jiffies + +msecs_to_jiffies (sysctl_sched_wake_affine_interval); +} +static int wake_affine (struct sched_domain *sd,struct task_struct *p,int sync) {s64 this_load, load;@@ -3096,6 +3112,9 @@ static int wake_affine(struct sched_domain *sd, struct task_struct *p, int sync)unsigned long weight;int balanced; +if (wake_affine_throttled (p)) +return 0 ; + idx = sd->wake_idx; this_cpu =smp_processor_id (); prev_cpu =task_cpu (p);@@ -3167,6 +3186,18 @@ static int wake_affine(struct sched_domain *sd, struct task_struct *p, int sync)return 1 ; } + +/* +* wake_affine() stuff try to pull wakee to the cpu +* around waker, this will benefit us if the data +* cached on waker cpu is hot for wakee, or the extreme +* ping-pong case. +* +* However, do such blindly work too frequently will +* cause regression to some workload, thus, each time +* when wake_affine() failed, throttle it for a while. +*/ +wake_affine_throttle (p);return 0 ; }kernel/sysctl.c +10
-0@@ -437,6 +437,16 @@ static struct ctl_table kern_table[] = { .extra1 = &one, },#endif +#ifdef CONFIG_SMP + { + .procname = "sched_wake_affine_interval", + .data = &sysctl_sched_wake_affine_interval, + .maxlen = sizeof(unsigned int), + .mode = 0644, + .proc_handler = proc_dointvec_minmax, + .extra1 = &zero, + }, +#endif #ifdef CONFIG_PROVE_LOCKING {Quoted 9 lines
kernel/sysctl.c static struct ctl_table kern_table[] = { · line 452
Quoted 2 lines
kernel/sysctl.c +0
-0.procname = "prove_locking",--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Quoted 13 lines · Michael Wang
On 04/10/2013 11:30 AM, Michael Wang wrote:
Now long time has been passed since the first version, I'd like to do some summary about current states:
On a 12 cpu box with tip 3.9.0-rc7, test show that:
1. remove wake-affine stuff cause regression on hackbench (could be 15%).
2. reserve wake-affine stuff cause regression on pgbench (could be 45%).And since neither remove or reserve is acceptable, this patch introduced a knob to throttle wake-affine stuff.
Quoted 7 lines
By turning the knob, we could benefit both hackbench and pgbench, or sacrifice one to significantly benefit another.
All similar workload which prefer or hate wake-affine behaviour could been cared.
If this approach caused any concerns, please let me know ;-)
I've just tested tip 3.10.0-rc1, it inherit the same issue from 3.9.0, pgbench was sacrificed and we still don't have any way to stop the damage...
Regards,
Michael Wang
Quoted 167 lines
Regards,
Michael WangQuoted 162 lines
Recently testing show that wake-affine stuff cause regression on pgbench, the hiding rat was finally catched out.
wake-affine stuff is always trying to pull wakee close to waker, by theory, this will benefit us if waker's cpu cached hot data for wakee, or the extreme ping-pong case.
However, the whole stuff is somewhat blindly, load balance is the only factor to be guaranteed, and since the stuff itself is time-consuming, some workload suffered, pgbench is just the one who has been found.
Thus, throttle the wake-affine stuff for such workload is necessary.
This patch introduced a new knob '
sysctl_sched_wake_affine_interval' with the default value 1ms (default minimum balance interval), which means wake-affine will keep silent for 1ms after it returned false.By turning the new knob, those workload who suffered will have the chance to stop the regression.
Test:
Test with 12 cpu X86 server and tip 3.9.0-rc2. default base 1ms interval 10ms interval 100ms interval | db_size | clients | tps | | tps | | tps | | tps | +---------+---------+-------+- +-------+ +-------+ +-------+ | 21 MB | 1 | 10572 | | 10804 | | 10802 | | 10801 | | 21 MB | 2 | 21275 | | 21533 | | 21400 | | 21498 | | 21 MB | 4 | 41866 | | 42158 | | 42410 | | 42306 | | 21 MB | 8 | 53931 | | 55796 | | 58608 | +8.67% | 59916 | +11.10% | 21 MB | 12 | 50956 | | 52266 | | 54586 | +7.12% | 55982 | +9.86% | 21 MB | 16 | 49911 | | 52862 | +5.91% | 55668 | +11.53% | 57255 | +14.71% | 21 MB | 24 | 46046 | | 48820 | +6.02% | 54269 | +17.86% | 58113 | +26.21% | 21 MB | 32 | 43405 | | 46635 | +7.44% | 53690 | +23.70% | 57729 | +33.00% | 7483 MB | 1 | 7734 | | 8013 | | 8046 | | 7879 | | 7483 MB | 2 | 19375 | | 19459 | | 19448 | | 19421 | | 7483 MB | 4 | 37408 | | 37780 | | 37937 | | 37819 | | 7483 MB | 8 | 49033 | | 50389 | | 51636 | +5.31% | 52294 | +6.65% | 7483 MB | 12 | 45525 | | 47794 | +4.98% | 49828 | +9.45% | 50571 | +11.08% | 7483 MB | 16 | 45731 | | 47921 | +4.79% | 50203 | +9.78% | 52033 | +13.78% | 7483 MB | 24 | 41533 | | 44301 | +6.67% | 49697 | +19.66% | 53833 | +29.62% | 7483 MB | 32 | 36370 | | 38301 | +5.31% | 48146 | +32.38% | 52795 | +45.16% | 15 GB | 1 | 7576 | | 7926 | | 7722 | | 7969 | | 15 GB | 2 | 19157 | | 19284 | | 19294 | | 19304 | | 15 GB | 4 | 37285 | | 37539 | | 37281 | | 37508 | | 15 GB | 8 | 48718 | | 49176 | | 50836 | +4.35% | 51239 | +5.17% | 15 GB | 12 | 45167 | | 47180 | +4.45% | 49206 | +8.94% | 50126 | +10.98% | 15 GB | 16 | 45270 | | 47293 | +4.47% | 49638 | +9.65% | 51748 | +14.31% | 15 GB | 24 | 40984 | | 43366 | +5.81% | 49356 | +20.43% | 53157 | +29.70% | 15 GB | 32 | 35918 | | 37632 | +4.77% | 47923 | +33.42% | 52241 | +45.45%Suggested-by Peter Zijlstra <[email protected]>Signed-off-by Michael Wang <[email protected]>--- include/linux/sched.h | 5 +++++ kernel/sched/fair.c | 31 +++++++++++++++++++++++++++++++ kernel/sysctl.c | 10 ++++++++++ 3 files changed, 46 insertions(+), 0 deletions(-)include/linux/sched.h +5
-0@@ -1197,6 +1197,10 @@ enum perf_event_task_context { perf_nr_task_contexts, }; +#ifdef CONFIG_SMP +extern unsigned int sysctl_sched_wake_affine_interval; +#endif +struct task_struct {volatile long state;/* -1 unrunnable, 0 runnable, >0 stopped */ void *stack;@@ -1207,6 +1211,7 @@ struct task_struct {#ifdef CONFIG_SMP struct llist_node wake_entry;int on_cpu; +unsigned long next_wake_affine;#endif int on_rq;kernel/sched/fair.c +31
-0@@ -3087,6 +3087,22 @@ static inline unsigned long effective_load(struct task_group *tg, int cpu,#endif +/* +* Default is 1ms, to prevent the wake_affine() stuff working too frequently. +*/ +unsigned int sysctl_sched_wake_affine_interval =1U ; + +static inline int wake_affine_throttled (struct task_struct *p) +{ +return time_before (jiffies, p->next_wake_affine); +} + +static inline void wake_affine_throttle (struct task_struct *p) +{ + p->next_wake_affine = jiffies + +msecs_to_jiffies (sysctl_sched_wake_affine_interval); +} +static int wake_affine (struct sched_domain *sd,struct task_struct *p,int sync) {s64 this_load, load;@@ -3096,6 +3112,9 @@ static int wake_affine(struct sched_domain *sd, struct task_struct *p, int sync)unsigned long weight;int balanced; +if (wake_affine_throttled (p)) +return 0 ; + idx = sd->wake_idx; this_cpu =smp_processor_id (); prev_cpu =task_cpu (p);@@ -3167,6 +3186,18 @@ static int wake_affine(struct sched_domain *sd, struct task_struct *p, int sync)return 1 ; } + +/* +* wake_affine() stuff try to pull wakee to the cpu +* around waker, this will benefit us if the data +* cached on waker cpu is hot for wakee, or the extreme +* ping-pong case. +* +* However, do such blindly work too frequently will +* cause regression to some workload, thus, each time +* when wake_affine() failed, throttle it for a while. +*/ +wake_affine_throttle (p);return 0 ; }kernel/sysctl.c +10
-0@@ -437,6 +437,16 @@ static struct ctl_table kern_table[] = { .extra1 = &one, },#endif +#ifdef CONFIG_SMP + { + .procname = "sched_wake_affine_interval", + .data = &sysctl_sched_wake_affine_interval, + .maxlen = sizeof(unsigned int), + .mode = 0644, + .proc_handler = proc_dointvec_minmax, + .extra1 = &zero, + }, +#endif #ifdef CONFIG_PROVE_LOCKING { .procname = "prove_locking",
Quoted 48 lines · Michael Wang
Log since RFC:
1. Throttle only when wake-affine failed. (thanks to PeterZ)
2. Do throttle insidewake_affine(). (thanks to PeterZ)
3. Other small fix.Recently testing show that wake-affine stuff cause regression on pgbench, the hiding rat was finally catched out.
wake-affine stuff is always trying to pull wakee close to waker, by theory, this will benefit us if waker's cpu cached hot data for wakee, or the extreme ping-pong case.
However, the whole stuff is somewhat blindly, load balance is the only factor to be guaranteed, and since the stuff itself is time-consuming, some workload suffered, pgbench is just the one who has been found.
Thus, throttle the wake-affine stuff for such workload is necessary.
This patch introduced a new knob '
sysctl_sched_wake_affine_interval' with the default value 1ms (default minimum balance interval), which means wake-affine will keep silent for 1ms after it returned false.By turning the new knob, those workload who suffered will have the chance to stop the regression.
Test:
Test with 12 cpu X86 server and tip 3.9.0-rc2. default base 1ms interval 10ms interval 100ms interval | db_size | clients | tps | | tps | | tps | | tps | +---------+---------+-------+- +-------+ +-------+ +-------+ | 21 MB | 1 | 10572 | | 10804 | | 10802 | | 10801 | | 21 MB | 2 | 21275 | | 21533 | | 21400 | | 21498 | | 21 MB | 4 | 41866 | | 42158 | | 42410 | | 42306 | | 21 MB | 8 | 53931 | | 55796 | | 58608 | +8.67% | 59916 | +11.10% | 21 MB | 12 | 50956 | | 52266 | | 54586 | +7.12% | 55982 | +9.86% | 21 MB | 16 | 49911 | | 52862 | +5.91% | 55668 | +11.53% | 57255 | +14.71% | 21 MB | 24 | 46046 | | 48820 | +6.02% | 54269 | +17.86% | 58113 | +26.21% | 21 MB | 32 | 43405 | | 46635 | +7.44% | 53690 | +23.70% | 57729 | +33.00% | 7483 MB | 1 | 7734 | | 8013 | | 8046 | | 7879 | | 7483 MB | 2 | 19375 | | 19459 | | 19448 | | 19421 | | 7483 MB | 4 | 37408 | | 37780 | | 37937 | | 37819 | | 7483 MB | 8 | 49033 | | 50389 | | 51636 | +5.31% | 52294 | +6.65% | 7483 MB | 12 | 45525 | | 47794 | +4.98% | 49828 | +9.45% | 50571 | +11.08% | 7483 MB | 16 | 45731 | | 47921 | +4.79% | 50203 | +9.78% | 52033 | +13.78% | 7483 MB | 24 | 41533 | | 44301 | +6.67% | 49697 | +19.66% | 53833 | +29.62% | 7483 MB | 32 | 36370 | | 38301 | +5.31% | 48146 | +32.38% | 52795 | +45.16%
Quoted 9 lines
| 15 GB | 1 | 7576 | | 7926 | | 7722 | | 7969 | | 15 GB | 2 | 19157 | | 19284 | | 19294 | | 19304 | | 15 GB | 4 | 37285 | | 37539 | | 37281 | | 37508 | | 15 GB | 8 | 48718 | | 49176 | | 50836 | +4.35% | 51239 | +5.17% | 15 GB | 12 | 45167 | | 47180 | +4.45% | 49206 | +8.94% | 50126 | +10.98% | 15 GB | 16 | 45270 | | 47293 | +4.47% | 49638 | +9.65% | 51748 | +14.31% | 15 GB | 24 | 40984 | | 43366 | +5.81% | 49356 | +20.43% | 53157 | +29.70% | 15 GB | 32 | 35918 | | 37632 | +4.77% | 47923 | +33.42% | 52241 | +45.45%
Add cc list :)
Quoted 109 lines
Suggested-by Peter Zijlstra <[email protected]>Signed-off-by Michael Wang <[email protected]>--- include/linux/sched.h | 5 +++++ kernel/sched/fair.c | 31 +++++++++++++++++++++++++++++++ kernel/sysctl.c | 10 ++++++++++ 3 files changed, 46 insertions(+), 0 deletions(-)include/linux/sched.h +5
-0@@ -1197,6 +1197,10 @@ enum perf_event_task_context { perf_nr_task_contexts, }; +#ifdef CONFIG_SMP +extern unsigned int sysctl_sched_wake_affine_interval; +#endif +struct task_struct {volatile long state;/* -1 unrunnable, 0 runnable, >0 stopped */ void *stack;@@ -1207,6 +1211,7 @@ struct task_struct {#ifdef CONFIG_SMP struct llist_node wake_entry;int on_cpu; +unsigned long next_wake_affine;#endif int on_rq;kernel/sched/fair.c +31
-0@@ -3087,6 +3087,22 @@ static inline unsigned long effective_load(struct task_group *tg, int cpu,#endif +/* +* Default is 1ms, to prevent the wake_affine() stuff working too frequently. +*/ +unsigned int sysctl_sched_wake_affine_interval =1U ; + +static inline int wake_affine_throttled (struct task_struct *p) +{ +return time_before (jiffies, p->next_wake_affine); +} + +static inline void wake_affine_throttle (struct task_struct *p) +{ + p->next_wake_affine = jiffies + +msecs_to_jiffies (sysctl_sched_wake_affine_interval); +} +static int wake_affine (struct sched_domain *sd,struct task_struct *p,int sync) {s64 this_load, load;@@ -3096,6 +3112,9 @@ static int wake_affine(struct sched_domain *sd, struct task_struct *p, int sync)unsigned long weight;int balanced; +if (wake_affine_throttled (p)) +return 0 ; + idx = sd->wake_idx; this_cpu =smp_processor_id (); prev_cpu =task_cpu (p);@@ -3167,6 +3186,18 @@ static int wake_affine(struct sched_domain *sd, struct task_struct *p, int sync)return 1 ; } + +/* +* wake_affine() stuff try to pull wakee to the cpu +* around waker, this will benefit us if the data +* cached on waker cpu is hot for wakee, or the extreme +* ping-pong case. +* +* However, do such blindly work too frequently will +* cause regression to some workload, thus, each time +* when wake_affine() failed, throttle it for a while. +*/ +wake_affine_throttle (p);return 0 ; }kernel/sysctl.c +10
-0@@ -437,6 +437,16 @@ static struct ctl_table kern_table[] = { .extra1 = &one, },#endif +#ifdef CONFIG_SMP + { + .procname = "sched_wake_affine_interval", + .data = &sysctl_sched_wake_affine_interval, + .maxlen = sizeof(unsigned int), + .mode = 0644, + .proc_handler = proc_dointvec_minmax, + .extra1 = &zero, + }, +#endif #ifdef CONFIG_PROVE_LOCKING { .procname = "prove_locking",
PATCH v2 1/1 sched: wake-affine throttle
In reply to Michael WangLog since v1:
Add cc list.
Add more comments.
Tested on tip 3.10.0-rc1.
wake-affine stuff is always trying to pull wakee close to waker, by theory, this will benefit us if waker's cpu cached hot data for wakee, or the extreme ping-pong case, and testing show it could benefit hackbench 15% at most.
However, the whole feature is somewhat blindly, load balance is the only factor to be guaranteed, and since the stuff itself is time-consuming, some workload suffered, and testing show it could damage pgbench 41% at most.
The feature currently settled in mainline, which means the current scheduler force sacrificed some workloads to benefit others, that is definitely unfair.
Thus, this patch provide the way to throttle wake-affine stuff, in order to adjust the gain and loss according to demand.
The patch introduced a new knob 'sysctl_sched_wake_affine_interval' with the default value 1ms (default minimum balance interval), which means wake-affine will keep silent for 1ms after it's failure.
By turning the new knob, compared with mainline, which currently blindly using wake-affine, pgbench show 41% improvement at most.
Link:
Analysis from Mike Galbraith about the improvement: https://lkml.org/lkml/2013/4/11/54 Analysis about the reason of throttle after failed: https://lkml.org/lkml/2013/5/3/31
Test:
Test with 12 cpu X86 server and tip 3.10.0-rc1. default base 1ms interval 10ms interval 100ms interval | db_size | clients | tps | | tps | | tps | | tps | +---------+---------+-------+ +-------+ +-------+ +-------+ | 22 MB | 1 | 10828 | | 10850 | | 10795 | | 10845 | | 22 MB | 2 | 21434 | | 21469 | | 21463 | | 21455 | | 22 MB | 4 | 41563 | | 41826 | | 41789 | | 41779 | | 22 MB | 8 | 53451 | | 54917 | | 59250 | | 59097 | | 22 MB | 12 | 48681 | | 50454 | | 53248 | | 54881 | | 22 MB | 16 | 46352 | | 49627 | +7.07% | 54029 | +16.56% | 55935 | +20.67% | 22 MB | 24 | 44200 | | 46745 | +5.76% | 52106 | +17.89% | 57907 | +31.01% | 22 MB | 32 | 43567 | | 45264 | +3.90% | 51463 | +18.12% | 57122 | +31.11% | 7484 MB | 1 | 8926 | | 8959 | | 8765 | | 8682 | | 7484 MB | 2 | 19308 | | 19470 | | 19397 | | 19409 | | 7484 MB | 4 | 37269 | | 37501 | | 37552 | | 37470 | | 7484 MB | 8 | 47277 | | 48452 | | 51535 | | 52095 | | 7484 MB | 12 | 42815 | | 45347 | | 48478 | | 49256 | | 7484 MB | 16 | 40951 | | 44063 | +7.60% | 48536 | +18.52% | 51141 | +24.88% | 7484 MB | 24 | 37389 | | 39620 | +5.97% | 47052 | +25.84% | 52720 | +41.00% | 7484 MB | 32 | 36705 | | 38109 | +3.83% | 45932 | +25.14% | 51456 | +40.19% | 15 GB | 1 | 8642 | | 8850 | | 9092 | | 8560 | | 15 GB | 2 | 19256 | | 19285 | | 19362 | | 19322 | | 15 GB | 4 | 37114 | | 37131 | | 37221 | | 37257 | | 15 GB | 8 | 47120 | | 48053 | | 50845 | | 50923 | | 15 GB | 12 | 42386 | | 44748 | | 47868 | | 48875 | | 15 GB | 16 | 40624 | | 43414 | +6.87% | 48169 | +18.57% | 50814 | +25.08% | 15 GB | 24 | 37110 | | 39096 | +5.35% | 46594 | +25.56% | 52477 | +41.41% | 15 GB | 32 | 36252 | | 37316 | +2.94% | 45327 | +25.03% | 51217 | +41.28%
--- include/linux/sched.h | 5 +++++ kernel/sched/fair.c | 35 +++++++++++++++++++++++++++++++++++ kernel/sysctl.c | 10 ++++++++++ 3 files changed, 50 insertions(+), 0 deletions(-)
include/linux/sched.h +5 -0
@@ -1031,6 +1031,10 @@ enum perf_event_task_context { perf_nr_task_contexts, }; +#ifdef CONFIG_SMP +extern unsigned int sysctl_sched_wake_affine_interval; +#endif +struct task_struct {volatile long state;/* -1 unrunnable, 0 runnable, >0 stopped */ void *stack;
@@ -1041,6 +1045,7 @@ struct task_struct {#ifdef CONFIG_SMP struct llist_node wake_entry;int on_cpu; +unsigned long next_wake_affine;#endif int on_rq;
kernel/sched/fair.c +35 -0
@@ -3127,6 +3127,22 @@ static inline unsigned long effective_load(struct task_group *tg, int cpu,#endif +/* +* Default is 1ms, to prevent the wake_affine() stuff working too frequently. +*/ +unsigned int sysctl_sched_wake_affine_interval =1U ; + +static inline int wake_affine_throttled (struct task_struct *p) +{ +return time_before (jiffies, p->next_wake_affine); +} + +static inline void wake_affine_throttle (struct task_struct *p) +{ + p->next_wake_affine = jiffies + +msecs_to_jiffies (sysctl_sched_wake_affine_interval); +} +static int wake_affine (struct sched_domain *sd,struct task_struct *p,int sync) {s64 this_load, load;
@@ -3136,6 +3152,9 @@ static int wake_affine(struct sched_domain *sd, struct task_struct *p, int sync)unsigned long weight;int balanced; +if (wake_affine_throttled (p)) +return 0 ; + idx = sd->wake_idx; this_cpu =smp_processor_id (); prev_cpu =task_cpu (p);
@@ -3207,6 +3226,22 @@ static int wake_affine(struct sched_domain *sd, struct task_struct *p, int sync)return 1 ; } + +/* +* wake_affine() stuff try to pull wakee to the cpu +* around waker, this will benefit us if the data +* cached on waker cpu is hot for wakee, or the extreme +* ping-pong case. +* +* However, do such blindly work too frequently will +* cause regression to some workload, thus, each time +* when wake_affine() failed, throttle it for a while. +* +* Throttle when failed is supposed to make the interval +* cover more failures, since failed wake_affine() +* is nothing but wasting cpu cycles. +*/ +wake_affine_throttle (p);return 0 ; }
kernel/sysctl.c +10 -0
@@ -436,6 +436,16 @@ static struct ctl_table kern_table[] = { .extra1 = &one, },#endif +#ifdef CONFIG_SMP + { + .procname = "sched_wake_affine_interval", + .data = &sysctl_sched_wake_affine_interval, + .maxlen = sizeof(unsigned int), + .mode = 0644, + .proc_handler = proc_dointvec_minmax, + .extra1 = &zero, + }, +#endif #ifdef CONFIG_PROVE_LOCKING { .procname = "prove_locking",
Signature · 1 line
1.7.4.1
Quoted 2 lines · Michael Wang
CC Mike Galbraith <[email protected]>CC Alex Shi <[email protected]>
You can change CC to
Signature · 2 lines
Thanks
AlexQuoted 6 lines · Alex Shi
Quoted 2 lines · Michael Wang
CC Mike Galbraith <[email protected]>CC Alex Shi <[email protected]>You can change CC to
Reviewed-by Alex Shi <[email protected]>
Got it, thanks for your review :)
Regards,
Michael Wang
Quoted 1 line
Peter Zijlstra
maintainerQuoted 61 lines · Michael Wang
wake-affine stuff is always trying to pull wakee close to waker, by theory, this will benefit us if waker's cpu cached hot data for wakee, or the extreme ping-pong case, and testing show it could benefit hackbench 15% at most.
However, the whole feature is somewhat blindly, load balance is the only factor to be guaranteed, and since the stuff itself is time-consuming, some workload suffered, and testing show it could damage pgbench 41% at most.
The feature currently settled in mainline, which means the current scheduler force sacrificed some workloads to benefit others, that is definitely unfair.
Thus, this patch provide the way to throttle wake-affine stuff, in order to adjust the gain and loss according to demand.
The patch introduced a new knob '
sysctl_sched_wake_affine_interval' with the default value 1ms (default minimum balance interval), which means wake-affine will keep silent for 1ms after it's failure.By turning the new knob, compared with mainline, which currently blindly using wake-affine, pgbench show 41% improvement at most.
Link:
Analysis from Mike Galbraith about the improvement: https://lkml.org/lkml/2013/4/11/54 Analysis about the reason of throttle after failed: https://lkml.org/lkml/2013/5/3/31Test:
Test with 12 cpu X86 server and tip 3.10.0-rc1. default base 1ms interval 10ms interval 100ms interval | db_size | clients | tps | | tps | | tps | | tps | +---------+---------+-------+ +-------+ +-------+ +-------+ | 22 MB | 1 | 10828 | | 10850 | | 10795 | | 10845 | | 22 MB | 2 | 21434 | | 21469 | | 21463 | | 21455 | | 22 MB | 4 | 41563 | | 41826 | | 41789 | | 41779 | | 22 MB | 8 | 53451 | | 54917 | | 59250 | | 59097 | | 22 MB | 12 | 48681 | | 50454 | | 53248 | | 54881 | | 22 MB | 16 | 46352 | | 49627 | +7.07% | 54029 | +16.56% | 55935 | +20.67% | 22 MB | 24 | 44200 | | 46745 | +5.76% | 52106 | +17.89% | 57907 | +31.01% | 22 MB | 32 | 43567 | | 45264 | +3.90% | 51463 | +18.12% | 57122 | +31.11% | 7484 MB | 1 | 8926 | | 8959 | | 8765 | | 8682 | | 7484 MB | 2 | 19308 | | 19470 | | 19397 | | 19409 | | 7484 MB | 4 | 37269 | | 37501 | | 37552 | | 37470 | | 7484 MB | 8 | 47277 | | 48452 | | 51535 | | 52095 | | 7484 MB | 12 | 42815 | | 45347 | | 48478 | | 49256 | | 7484 MB | 16 | 40951 | | 44063 | +7.60% | 48536 | +18.52% | 51141 | +24.88% | 7484 MB | 24 | 37389 | | 39620 | +5.97% | 47052 | +25.84% | 52720 | +41.00% | 7484 MB | 32 | 36705 | | 38109 | +3.83% | 45932 | +25.14% | 51456 | +40.19% | 15 GB | 1 | 8642 | | 8850 | | 9092 | | 8560 | | 15 GB | 2 | 19256 | | 19285 | | 19362 | | 19322 | | 15 GB | 4 | 37114 | | 37131 | | 37221 | | 37257 | | 15 GB | 8 | 47120 | | 48053 | | 50845 | | 50923 | | 15 GB | 12 | 42386 | | 44748 | | 47868 | | 48875 | | 15 GB | 16 | 40624 | | 43414 | +6.87% | 48169 | +18.57% | 50814 | +25.08% | 15 GB | 24 | 37110 | | 39096 | +5.35% | 46594 | +25.56% | 52477 | +41.41% | 15 GB | 32 | 36252 | | 37316 | +2.94% | 45327 | +25.03% | 51217 | +41.28%
Quoted 6 lines
CC Ingo Molnar <[email protected]>CC Peter Zijlstra <[email protected]>CC Mike Galbraith <[email protected]>CC Alex Shi <[email protected]>Suggested-by Peter Zijlstra <[email protected]>Signed-off-by Michael Wang <[email protected]>
So I utterly hate this patch. I hate it worse than your initial buddy patch :/
And I know its got a Suggested-by there; but that was when you led me to believe that wake_affine() itself was expensive to run; its not, its the result of those runs you don't like.
While we have a ton (too many to be sure) scheduler tunables, users shouldn't ever need to actually touch those. Its just that every time we have to make a random choice its as easy to make it a debug knob as to hardcode it.
The problem with this patch is that users _have_ to frob knobs and while doing so potentially wreck other workloads.
To make it worse, the knob isn't anything fundamental, its a random hack.
So I would really either improve the smarts of wake_affine, with for example your wake buddy relation thing (and simply exempt [Soft]IRQs) or kill wake_affine and be done with it.
Either avenue has the risk of regressing some workload, but at least when that happens (and people report it) we'll have a counter-example to learn from and incorporate.
On 05/22/2013 04:49 PM, Peter Zijlstra wrote:
[snip]
Quoted 7 lines
Quoted 7 lines
CC Ingo Molnar <[email protected]>CC Peter Zijlstra <[email protected]>CC Mike Galbraith <[email protected]>CC Alex Shi <[email protected]>Suggested-by Peter Zijlstra <[email protected]>Signed-off-by Michael Wang <[email protected]>
Thanks for your reply, I've looking forward it for a long time...
Quoted 3 lines
So I utterly hate this patch. I hate it worse than your initial buddy patch :/
Then we nuke it, and figure out the better one ;-)
Quoted 4 lines
And I know its got a Suggested-by there; but that was when you led me to believe that
wake_affine()itself was expensive to run; its not, its the result of those runs you don't like.
Both are the reason, it's just the game between gain & lost & cost, your suggestion definitely is a good choice, otherwise I won't pay time on it, and I will call it's the best one if we are searching for a quick fix.
Quoted 11 lines
While we have a ton (too many to be sure) scheduler tunables, users shouldn't ever need to actually touch those. Its just that every time we have to make a random choice its as easy to make it a debug knob as to hardcode it.
The problem with this patch is that users _have_ to frob knobs and while doing so potentially wreck other workloads.
To make it worse, the knob isn't anything fundamental, its a random hack.
So we discard.
Quoted 4 lines
So I would really either improve the smarts of
wake_affine, with for example your wake buddy relation thing (and simply exempt [Soft]IRQs) or killwake_affineand be done with it.
No kill...we show mercy, I will back to the wakeup-buddy and let's forgot the IRQ case temporarily unless some regression report appear.
Quoted 4 lines
Either avenue has the risk of regressing some workload, but at least when that happens (and people report it) we'll have a counter-example to learn from and incorporate.
I've not test the hackbench with wakeup-buddy before, will do it this time, I suppose the 15% illegal income will suffered, anyway, it's illegal :)
Regards,
Michael Wang
Quoted 6 lines
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Quoted 3 lines · Michael Wang
I've not test the hackbench with wakeup-buddy before, will do it this time, I suppose the 15% illegal income will suffered, anyway, it's illegal :)
On a 4 socket 40 core (+SMT) box, hackbench wasn't too happy.
defaultx = wakeup buddy
tbench 30 localhost
1 5 10 20 40 80 160 320 3.10.0-default 188.13 953.03 1860.03 3180.93 5378.34 10826.06 11342.90 11651.30 3.10.0-defaultx 187.26 934.55 1859.30 3160.29 5016.35 12477.15 12567.05 12068.47
hackbench -l 400 -g 400
3.10.0-default Time: 3.919 4.250 4.116 3.10.0-defaultx Time: 9.074 10.985 9.849
aim7 compute
AIM Multiuser Benchmark - Suite VII "1.1" 3.10.0-default AIM Multiuser Benchmark - Suite VII "1.1" 3.10.0-defaultx
Tasks Jobs/Min JTI Real CPU Jobs/sec/task Tasks Jobs/Min JTI Real CPU Jobs/sec/task 1 428.0 100 14.2 4.1 7.1328 1 428.9 100 14.1 4.1 7.1479 1 417.9 100 14.5 4.1 6.9655 1 430.4 100 14.1 4.0 7.1733 1 427.7 100 14.2 4.2 7.1277 1 424.7 100 14.3 4.2 7.0778 5 2350.7 99 12.9 13.8 7.8355 5 2156.6 99 14.1 19.8 7.1886 5 2422.1 99 12.5 12.1 8.0735 5 2155.0 99 14.1 19.7 7.1835 5 2189.3 98 13.8 18.3 7.2977 5 2108.6 99 14.4 21.4 7.0285 10 4515.6 93 13.4 27.6 7.5261 10 4529.1 96 13.4 29.5 7.5486 10 4708.6 96 12.9 24.3 7.8477 10 4597.9 96 13.2 26.9 7.6631 10 4636.6 96 13.1 25.7 7.7276 10 5197.3 98 11.7 14.8 8.6621 20 8053.2 95 15.1 78.1 6.7110 20 9431.9 98 12.8 49.1 7.8599 20 8250.5 92 14.7 67.5 6.8754 20 7973.7 97 15.2 93.4 6.6447 20 8178.1 97 14.8 78.5 6.8151 20 8145.2 95 14.9 78.4 6.7876 40 17413.8 94 13.9 88.6 7.2557 40 16312.2 92 14.9 115.6 6.7968 40 16775.1 93 14.5 111.6 6.9896 40 17070.4 94 14.2 110.6 7.1127 40 16031.7 93 15.1 147.1 6.6799 40 17578.0 94 13.8 96.9 7.3241 80 33666.7 95 14.4 138.4 7.0139 80 33854.7 96 14.3 177.9 7.0531 80 33949.6 97 14.3 128.0 7.0728 80 34164.9 96 14.2 146.4 7.1177 80 35752.2 96 13.6 159.1 7.4484 80 33807.5 96 14.3 127.6 7.0432 160 74814.8 98 13.0 149.8 7.7932 160 75162.8 98 12.9 148.6 7.8295 160 74015.3 97 13.1 149.5 7.7099 160 74642.0 98 13.0 168.2 7.7752 160 73621.9 98 13.2 146.3 7.6689 160 75572.9 98 12.8 163.6 7.8722 320 139210.3 96 13.9 280.1 7.2505 320 139010.8 97 14.0 282.4 7.2401 320 135135.9 96 14.3 277.8 7.0383 320 139611.2 96 13.9 282.2 7.2714 320 139110.5 96 13.9 280.4 7.2453 320 138514.3 97 14.0 281.0 7.2143 640 223538.9 98 17.4 577.2 5.8213 640 224704.5 95 17.3 567.3 5.8517 640 224055.5 97 17.3 575.7 5.8348 640 222385.3 95 17.4 580.1 5.7913 640 225488.4 96 17.2 566.3 5.8721 640 225882.4 93 17.2 563.0 5.8824
Quoted 6 lines · Mike Galbraith
Quoted 3 lines · Michael Wang
I've not test the hackbench with wakeup-buddy before, will do it this time, I suppose the 15% illegal income will suffered, anyway, it's illegal :)
Thanks, Mike, these are very helpful data ;-)
Quoted 1 line
On a 4 socket 40 core (+SMT) box, hackbench wasn't too happy.
I'm not sure whether this is caused by the SOFT IRQ case, I have some idea in mind which may could help to solve that issue, but if this huge regression is caused by other reasons...
Anyway, I will rework on the idea and let's see whether we could make things better :)
Regards,
Michael Wang
Quoted 51 lines
defaultx = wakeup buddy
tbench 30 localhost
1 5 10 20 40 80 160 320 3.10.0-default 188.13 953.03 1860.03 3180.93 5378.34 10826.06 11342.90 11651.30 3.10.0-defaultx 187.26 934.55 1859.30 3160.29 5016.35 12477.15 12567.05 12068.47hackbench -l 400 -g 400
3.10.0-default Time: 3.919 4.250 4.116 3.10.0-defaultx Time: 9.074 10.985 9.849aim7 compute
AIM Multiuser Benchmark - Suite VII "1.1" 3.10.0-default AIM Multiuser Benchmark - Suite VII "1.1" 3.10.0-defaultxTasks Jobs/Min JTI Real CPU Jobs/sec/task Tasks Jobs/Min JTI Real CPU Jobs/sec/task 1 428.0 100 14.2 4.1 7.1328 1 428.9 100 14.1 4.1 7.1479 1 417.9 100 14.5 4.1 6.9655 1 430.4 100 14.1 4.0 7.1733 1 427.7 100 14.2 4.2 7.1277 1 424.7 100 14.3 4.2 7.0778 5 2350.7 99 12.9 13.8 7.8355 5 2156.6 99 14.1 19.8 7.1886 5 2422.1 99 12.5 12.1 8.0735 5 2155.0 99 14.1 19.7 7.1835 5 2189.3 98 13.8 18.3 7.2977 5 2108.6 99 14.4 21.4 7.0285 10 4515.6 93 13.4 27.6 7.5261 10 4529.1 96 13.4 29.5 7.5486 10 4708.6 96 12.9 24.3 7.8477 10 4597.9 96 13.2 26.9 7.6631 10 4636.6 96 13.1 25.7 7.7276 10 5197.3 98 11.7 14.8 8.6621 20 8053.2 95 15.1 78.1 6.7110 20 9431.9 98 12.8 49.1 7.8599 20 8250.5 92 14.7 67.5 6.8754 20 7973.7 97 15.2 93.4 6.6447 20 8178.1 97 14.8 78.5 6.8151 20 8145.2 95 14.9 78.4 6.7876 40 17413.8 94 13.9 88.6 7.2557 40 16312.2 92 14.9 115.6 6.7968 40 16775.1 93 14.5 111.6 6.9896 40 17070.4 94 14.2 110.6 7.1127 40 16031.7 93 15.1 147.1 6.6799 40 17578.0 94 13.8 96.9 7.3241 80 33666.7 95 14.4 138.4 7.0139 80 33854.7 96 14.3 177.9 7.0531 80 33949.6 97 14.3 128.0 7.0728 80 34164.9 96 14.2 146.4 7.1177 80 35752.2 96 13.6 159.1 7.4484 80 33807.5 96 14.3 127.6 7.0432 160 74814.8 98 13.0 149.8 7.7932 160 75162.8 98 12.9 148.6 7.8295 160 74015.3 97 13.1 149.5 7.7099 160 74642.0 98 13.0 168.2 7.7752 160 73621.9 98 13.2 146.3 7.6689 160 75572.9 98 12.8 163.6 7.8722 320 139210.3 96 13.9 280.1 7.2505 320 139010.8 97 14.0 282.4 7.2401 320 135135.9 96 14.3 277.8 7.0383 320 139611.2 96 13.9 282.2 7.2714 320 139110.5 96 13.9 280.4 7.2453 320 138514.3 97 14.0 281.0 7.2143 640 223538.9 98 17.4 577.2 5.8213 640 224704.5 95 17.3 567.3 5.8517 640 224055.5 97 17.3 575.7 5.8348 640 222385.3 95 17.4 580.1 5.7913 640 225488.4 96 17.2 566.3 5.8721 640 225882.4 93 17.2 563.0 5.8824--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Quoted 7 lines · Mike Galbraith
Quoted 3 lines · Michael Wang
I've not test the hackbench with wakeup-buddy before, will do it this time, I suppose the 15% illegal income will suffered, anyway, it's illegal :)
On a 4 socket 40 core (+SMT) box, hackbench wasn't too happy.
I've done more test and now I got the reason of regression...
The writer and reader in hackbench is N:N, prev writer will write all the fd then switch to next writer and repeat the same work, so it's impossible to setup the buddy relationship by just record the last one, and we have to record all the waker/wakee in history, but that means unacceptable memory overhead...
So this buddy idea seems to be bad...
I think a better way may should be allowing pull in most time, but filter the very bad cases carefully.
For workload like pgbench, we actually just need to avoid pull if that will damage the 'mother' thread, which is busy and be relied by many 'child'.
I will send out a new implementation, let's see whether it could solve the problems ;-)
Regards,
Michael Wang
Quoted 51 lines
defaultx = wakeup buddy
tbench 30 localhost
1 5 10 20 40 80 160 320 3.10.0-default 188.13 953.03 1860.03 3180.93 5378.34 10826.06 11342.90 11651.30 3.10.0-defaultx 187.26 934.55 1859.30 3160.29 5016.35 12477.15 12567.05 12068.47hackbench -l 400 -g 400
3.10.0-default Time: 3.919 4.250 4.116 3.10.0-defaultx Time: 9.074 10.985 9.849aim7 compute
AIM Multiuser Benchmark - Suite VII "1.1" 3.10.0-default AIM Multiuser Benchmark - Suite VII "1.1" 3.10.0-defaultxTasks Jobs/Min JTI Real CPU Jobs/sec/task Tasks Jobs/Min JTI Real CPU Jobs/sec/task 1 428.0 100 14.2 4.1 7.1328 1 428.9 100 14.1 4.1 7.1479 1 417.9 100 14.5 4.1 6.9655 1 430.4 100 14.1 4.0 7.1733 1 427.7 100 14.2 4.2 7.1277 1 424.7 100 14.3 4.2 7.0778 5 2350.7 99 12.9 13.8 7.8355 5 2156.6 99 14.1 19.8 7.1886 5 2422.1 99 12.5 12.1 8.0735 5 2155.0 99 14.1 19.7 7.1835 5 2189.3 98 13.8 18.3 7.2977 5 2108.6 99 14.4 21.4 7.0285 10 4515.6 93 13.4 27.6 7.5261 10 4529.1 96 13.4 29.5 7.5486 10 4708.6 96 12.9 24.3 7.8477 10 4597.9 96 13.2 26.9 7.6631 10 4636.6 96 13.1 25.7 7.7276 10 5197.3 98 11.7 14.8 8.6621 20 8053.2 95 15.1 78.1 6.7110 20 9431.9 98 12.8 49.1 7.8599 20 8250.5 92 14.7 67.5 6.8754 20 7973.7 97 15.2 93.4 6.6447 20 8178.1 97 14.8 78.5 6.8151 20 8145.2 95 14.9 78.4 6.7876 40 17413.8 94 13.9 88.6 7.2557 40 16312.2 92 14.9 115.6 6.7968 40 16775.1 93 14.5 111.6 6.9896 40 17070.4 94 14.2 110.6 7.1127 40 16031.7 93 15.1 147.1 6.6799 40 17578.0 94 13.8 96.9 7.3241 80 33666.7 95 14.4 138.4 7.0139 80 33854.7 96 14.3 177.9 7.0531 80 33949.6 97 14.3 128.0 7.0728 80 34164.9 96 14.2 146.4 7.1177 80 35752.2 96 13.6 159.1 7.4484 80 33807.5 96 14.3 127.6 7.0432 160 74814.8 98 13.0 149.8 7.7932 160 75162.8 98 12.9 148.6 7.8295 160 74015.3 97 13.1 149.5 7.7099 160 74642.0 98 13.0 168.2 7.7752 160 73621.9 98 13.2 146.3 7.6689 160 75572.9 98 12.8 163.6 7.8722 320 139210.3 96 13.9 280.1 7.2505 320 139010.8 97 14.0 282.4 7.2401 320 135135.9 96 14.3 277.8 7.0383 320 139611.2 96 13.9 282.2 7.2714 320 139110.5 96 13.9 280.4 7.2453 320 138514.3 97 14.0 281.0 7.2143 640 223538.9 98 17.4 577.2 5.8213 640 224704.5 95 17.3 567.3 5.8517 640 224055.5 97 17.3 575.7 5.8348 640 222385.3 95 17.4 580.1 5.7913 640 225488.4 96 17.2 566.3 5.8721 640 225882.4 93 17.2 563.0 5.8824--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Quoted 9 lines · Michael Wang
Quoted 7 lines · Mike Galbraith
Quoted 3 lines · Michael Wang
I've not test the hackbench with wakeup-buddy before, will do it this time, I suppose the 15% illegal income will suffered, anyway, it's illegal :)
On a 4 socket 40 core (+SMT) box, hackbench wasn't too happy.
Quoted 7 lines
I've done more test and now I got the reason of regression...
The writer and reader in hackbench is N:N, prev writer will write all the fd then switch to next writer and repeat the same work, so it's impossible to setup the buddy relationship by just record the last one, and we have to record all the waker/wakee in history, but that means unacceptable memory overhead...
Yeah, that's why I was thinking we'd need a dinky/fast as hell FIFO of tokens or such to bind waker/wakee more or less reliably. Making such a scheme cheap enough could be hard.
Quoted 4 lines
So this buddy idea seems to be bad...
I think a better way may should be allowing pull in most time, but filter the very bad cases carefully.
Any way that is cheap, and fairly accurately recognizes when we're being stupid will help. First and foremost, it has to be dirt cheap :)
Quoted 3 lines
For workload like pgbench, we actually just need to avoid pull if that will damage the 'mother' thread, which is busy and be relied by many 'child'.
Yeah, 'mom' is the key player. If we can cheaply recognize mom, that should get us a generic improvement. Not as good as being able to recognize the size of her+brood as size changes, but better anyway.
-Mike
Quoted 30 lines · Mike Galbraith
Quoted 9 lines · Michael Wang
Quoted 7 lines · Mike Galbraith
Quoted 3 lines · Michael Wang
I've not test the hackbench with wakeup-buddy before, will do it this time, I suppose the 15% illegal income will suffered, anyway, it's illegal :)
On a 4 socket 40 core (+SMT) box, hackbench wasn't too happy.
Quoted 7 lines
I've done more test and now I got the reason of regression...
The writer and reader in hackbench is N:N, prev writer will write all the fd then switch to next writer and repeat the same work, so it's impossible to setup the buddy relationship by just record the last one, and we have to record all the waker/wakee in history, but that means unacceptable memory overhead...
Yeah, that's why I was thinking we'd need a dinky/fast as hell FIFO of tokens or such to bind waker/wakee more or less reliably. Making such a scheme cheap enough could be hard.
Quoted 4 lines
So this buddy idea seems to be bad...
I think a better way may should be allowing pull in most time, but filter the very bad cases carefully.
Any way that is cheap, and fairly accurately recognizes when we're being stupid will help. First and foremost, it has to be dirt cheap :)
Quoted 7 lines
Quoted 3 lines
For workload like pgbench, we actually just need to avoid pull if that will damage the 'mother' thread, which is busy and be relied by many 'child'.
Yeah, 'mom' is the key player. If we can cheaply recognize mom, that should get us a generic improvement. Not as good as being able to recognize the size of her+brood as size changes, but better anyway.
That's right, I'm trying to rely on the frequency of a task switching it's wakee in the new idea, it's really cheap and somewhat reliable, I appreciate if you could pay an eye on the new patch and let me know you opinion :)
Regards,
Michael Wang
Quoted 3 lines
-Mike
Quoted 35 lines · Michael Wang
Quoted 34 lines · Mike Galbraith
Quoted 9 lines · Michael Wang
Quoted 7 lines · Mike Galbraith
Quoted 3 lines · Michael Wang
I've not test the hackbench with wakeup-buddy before, will do it this time, I suppose the 15% illegal income will suffered, anyway, it's illegal :)
On a 4 socket 40 core (+SMT) box, hackbench wasn't too happy.
Quoted 7 lines
I've done more test and now I got the reason of regression...
The writer and reader in hackbench is N:N, prev writer will write all the fd then switch to next writer and repeat the same work, so it's impossible to setup the buddy relationship by just record the last one, and we have to record all the waker/wakee in history, but that means unacceptable memory overhead...
Yeah, that's why I was thinking we'd need a dinky/fast as hell FIFO of tokens or such to bind waker/wakee more or less reliably. Making such a scheme cheap enough could be hard.
Quoted 4 lines
So this buddy idea seems to be bad...
I think a better way may should be allowing pull in most time, but filter the very bad cases carefully.
Any way that is cheap, and fairly accurately recognizes when we're being stupid will help. First and foremost, it has to be dirt cheap :)
Quoted 3 lines
For workload like pgbench, we actually just need to avoid pull if that will damage the 'mother' thread, which is busy and be relied by many 'child'.
Quoted 8 lines
Quoted 3 lines
Yeah, 'mom' is the key player. If we can cheaply recognize mom, that should get us a generic improvement. Not as good as being able to recognize the size of her+brood as size changes, but better anyway.
That's right, I'm trying to rely on the frequency of a task switching it's wakee in the new idea, it's really cheap and somewhat reliable, I appreciate if you could pay an eye on the new patch and let me know you opinion :)
I'll feed it to my 40 core box, hopefully soon.
-Mike
Quoted 40 lines · Mike Galbraith
Quoted 39 lines · Michael Wang
Quoted 37 lines · Mike Galbraith
Quoted 9 lines · Michael Wang
Quoted 7 lines · Mike Galbraith
Quoted 3 lines · Michael Wang
I've not test the hackbench with wakeup-buddy before, will do it this time, I suppose the 15% illegal income will suffered, anyway, it's illegal :)
On a 4 socket 40 core (+SMT) box, hackbench wasn't too happy.
Quoted 7 lines
I've done more test and now I got the reason of regression...
The writer and reader in hackbench is N:N, prev writer will write all the fd then switch to next writer and repeat the same work, so it's impossible to setup the buddy relationship by just record the last one, and we have to record all the waker/wakee in history, but that means unacceptable memory overhead...
Yeah, that's why I was thinking we'd need a dinky/fast as hell FIFO of tokens or such to bind waker/wakee more or less reliably. Making such a scheme cheap enough could be hard.
Quoted 4 lines
So this buddy idea seems to be bad...
I think a better way may should be allowing pull in most time, but filter the very bad cases carefully.
Any way that is cheap, and fairly accurately recognizes when we're being stupid will help. First and foremost, it has to be dirt cheap :)
Quoted 3 lines
For workload like pgbench, we actually just need to avoid pull if that will damage the 'mother' thread, which is busy and be relied by many 'child'.
Yeah, 'mom' is the key player. If we can cheaply recognize mom, that should get us a generic improvement. Not as good as being able to recognize the size of her+brood as size changes, but better anyway.
Quoted 6 lines
Quoted 4 lines
That's right, I'm trying to rely on the frequency of a task switching it's wakee in the new idea, it's really cheap and somewhat reliable, I appreciate if you could pay an eye on the new patch and let me know you opinion :)
I'll feed it to my 40 core box, hopefully soon.
Thanks for that, wish it taste delicious ;-)
Regards,
Michael Wang
Quoted 3 lines
-Mike