smp 'nice' bias support breaks scheduler behavior
Con,
Quoted 1 line
[PATCH] sched: implement nice support across physical cpus on SMP
I don't see imbalance calculations in find_busiest_group() take prio_bias into account. This will result in wrong imbalance value and will cause issues.
For example on a DP system with HT, if there are three runnable processes (simple infinite loop with same nice value), this patch is resulting in bouncing of these 3 processes from one processor to another...Lets assume
if the 3 processes are scheduled as 2 in package-0 and 1 in package1.. Now when the busy processor on package-1 does load balance and as imbalance doesn't take "prio_bias" into account, this will kick active load balance on package-0.. And this is continuing for ever, resulting in bouncing from one processor to another.
Even when the system is completely loaded and if there is an imbalance, this patch causes wrong imabalance counts and cause unoptimized movements.
Do you want to look into this and post a patch for 2.6.16?
thanks,
suresh
Quoted 7 lines · Suresh B
Con,
Quoted 1 line
[PATCH] sched: implement nice support across physical cpus on SMP
I don't see imbalance calculations in
find_busiest_group()takeprio_biasinto account. This will result in wrong imbalance value and will cause issues.
in 2.6.16-rc1:
find_busiest_group(....
load = __target_load(i, load_idx, idle); else load = __source_load(i, load_idx, idle);
where __target_load and __source_load is where we take into account prio_bias.
I'm not sure which code you're looking at, but Peter Williams is working on rewriting the smp nice balancing code in -mm at the moment so that is quite different from current linus tree.
Con
Quoted 18 lines
For example on a DP system with HT, if there are three runnable processes (simple infinite loop with same nice value), this patch is resulting in bouncing of these 3 processes from one processor to another...Lets assume if the 3 processes are scheduled as 2 in package-0 and 1 in package1.. Now when the busy processor on package-1 does load balance and as imbalance doesn't take "
prio_bias" into account, this will kick active load balance on package-0.. And this is continuing for ever, resulting in bouncing from one processor to another.Even when the system is completely loaded and if there is an imbalance, this patch causes wrong imabalance counts and cause unoptimized movements.
Do you want to look into this and post a patch for 2.6.16?
thanks,
suresh
Quoted 17 lines · Con Kolivas
Quoted 8 lines · Suresh B
Con,
Quoted 1 line
[PATCH] sched: implement nice support across physical cpus on SMP
I don't see imbalance calculations in
find_busiest_group()takeprio_biasinto account. This will result in wrong imbalance value and will cause issues.in 2.6.16-rc1:
find_busiest_group(....
Quoted 8 lines
load = __target_load(i, load_idx, idle); else load = __source_load(i, load_idx, idle);where
__target_loadand__source_loadis where we take into accountprio_bias.I'm not sure which code you're looking at, but Peter Williams is working on rewriting the smp nice balancing code in -mm at the moment so that is quite different from current linus tree.
Yes, indeed. And it would be very helpful if people interested in this topic (and that have test suites designed to test whether "niceness" is being well balanced across CPUs) could test it. This is especially the case for larger systems as I do not have ready access for testing on them.
Quoted 11 lines
Quoted 11 lines
For example on a DP system with HT, if there are three runnable processes (simple infinite loop with same nice value), this patch is resulting in bouncing of these 3 processes from one processor to another...Lets assume if the 3 processes are scheduled as 2 in package-0 and 1 in package1.. Now when the busy processor on package-1 does load balance and as imbalance doesn't take "
prio_bias" into account, this will kick active load balance on package-0.. And this is continuing for ever, resulting in bouncing from one processor to another.
Quoted 5 lines
Quoted 5 lines
Even when the system is completely loaded and if there is an imbalance, this patch causes wrong imabalance counts and cause unoptimized movements.
Do you want to look into this and post a patch for 2.6.16?
Thanks,
Peter
Signature · 4 lines
Peter Williams [email protected] "Learning, n. The kind of ignorance distinguishing the studious." -- Ambrose Bierce
Quoted 9 lines · Peter Williams
Quoted 8 lines · Con Kolivas
Quoted 5 lines · Suresh B
Quoted 1 line
[PATCH] sched: implement nice support across physical cpus on SMP
I don't see imbalance calculations in
find_busiest_group()takeprio_biasinto account. This will result in wrong imbalance value and will cause issues.in 2.6.16-rc1:
Quoted 7 lines
Quoted 7 lines
find_busiest_group(....
load = __target_load(i, load_idx, idle); else load = __source_load(i, load_idx, idle);where
__target_loadand__source_loadis where we take into accountprio_bias.
We take that into consideration only while calculating the loads.. But we don't scale it down while calculating imbalance, resulting in the problem I mentioned.
Quoted 5 lines
Quoted 5 lines
I'm not sure which code you're looking at, but Peter Williams is working on rewriting the smp nice balancing code in -mm at the moment so that is quite different from current linus tree.
Peters changes in -mm fix this issue. Will this be pushed to Linus tree before 2.6.16 comes out?
Quoted 5 lines
Yes, indeed. And it would be very helpful if people interested in this topic (and that have test suites designed to test whether "niceness" is being well balanced across CPUs) could test it. This is especially the case for larger systems as I do not have ready access for testing on them.
I don't have any test suites for testing "niceness". But I can def check more to make sure that it doesn't cause any regression :)
thanks,
suresh
Quoted 17 lines
Quoted 17 lines
Quoted 17 lines
For example on a DP system with HT, if there are three runnable processes (simple infinite loop with same nice value), this patch is resulting in bouncing of these 3 processes from one processor to another...Lets assume if the 3 processes are scheduled as 2 in package-0 and 1 in package1.. Now when the busy processor on package-1 does load balance and as imbalance doesn't take "
prio_bias" into account, this will kick active load balance on package-0.. And this is continuing for ever, resulting in bouncing from one processor to another.Even when the system is completely loaded and if there is an imbalance, this patch causes wrong imabalance counts and cause unoptimized movements.
Do you want to look into this and post a patch for 2.6.16?
Quoted 7 lines · Suresh B
Quoted 4 lines
Quoted 3 lines · Con Kolivas
I'm not sure which code you're looking at, but Peter Williams is working on rewriting the smp nice balancing code in -mm at the moment so that is quite different from current linus tree.
Peters changes in -mm fix this issue. Will this be pushed to Linus tree before 2.6.16 comes out?
We planned to push it for 2.6.16 but issues showed up, which Peter has since addressed - but this means the code has not had enough testing to go into 2.6.16 - so it is likely to be in 2.6.17.
Cheers,
Con
Quoted 3 lines · Con Kolivas
We planned to push it for 2.6.16 but issues showed up, which Peter has since addressed - but this means the code has not had enough testing to go into 2.6.16 - so it is likely to be in 2.6.17.
Then for 2.6.16, I would like to see smp nice handling patch backed out.
Do you agree?
thanks,
suresh
Quoted 8 lines · Suresh B
Quoted 3 lines · Con Kolivas
We planned to push it for 2.6.16 but issues showed up, which Peter has since addressed - but this means the code has not had enough testing to go into 2.6.16 - so it is likely to be in 2.6.17.
Then for 2.6.16, I would like to see smp nice handling patch backed out.
Do you agree?
It's not my decision to keep Peter's patch out of mainline. If you can make a strong enough case for it then Linus will merge it up even though it's after rc1. Otherwise I'll let Ingo decide on whether to pull the current implementation or not - you're saying that with the one thing you described that misbehaves that it is doing more harm than fixing smp nice handling.
Con
Quoted 3 lines · Con Kolivas
It's not my decision to keep Peter's patch out of mainline. If you can make a strong enough case for it then Linus will merge it up even though it's after rc1.
I don't want to push Peters patch to 2.6.16, as I haven't tested much.
Quoted 3 lines
Otherwise I'll let Ingo decide on whether to pull the current implementation or not - you're saying that with the one thing you described that misbehaves that it is doing more harm than fixing smp nice handling.
Are we sure that it really fixes smp nice handling? Its not just one scenario(bouncing processes on a lightly loaded system), I am talking about. Imbalance calculations will be wrong even on a completely loaded system.. Are you sure that there are no perf regressions with your patch..
Sorry for commenting on this patch so late.. I was on a very long vacation. I think it is safe to back that out for 2.6.16 and do more work and get it in 2.6.17.
thanks,
suresh
Quoted 12 lines · Suresh B
Quoted 3 lines · Con Kolivas
It's not my decision to keep Peter's patch out of mainline. If you can make a strong enough case for it then Linus will merge it up even though it's after rc1.
I don't want to push Peters patch to 2.6.16, as I haven't tested much.
Quoted 4 lines
Otherwise I'll let Ingo decide on whether to pull the current implementation or not - you're saying that with the one thing you described that misbehaves that it is doing more harm than fixing smp nice handling.
Quoted 4 lines
Are we sure that it really fixes smp nice handling? Its not just one scenario(bouncing processes on a lightly loaded system), I am talking about. Imbalance calculations will be wrong even on a completely loaded system.. Are you sure that there are no perf regressions with your patch..
It was extensively tested for more than 3 months in the -mm tree. Early on there were accounting bugs in the code which I corrected and we saw no performance regression after that across a wide range of benchmarks and hardware configurations at the time thanks to M Bligh. (see test.kernel.org) Some were done on the osdl (STP) test bench showing no regression as well but the osdl infrastructure became pretty much unworkable not long after.
Quoted 3 lines
Sorry for commenting on this patch so late.. I was on a very long vacation. I think it is safe to back that out for 2.6.16 and do more work and get it in 2.6.17.
Well I have no emotional investment in the code, I just want to do what's right. In the absence of measurable throughput regressions and improvement in smp nice handling I don't believe we should back it out.
Cheers,
Con