sched/fair: Consider RT/IRQ pressure in capacity_spare_wake

capacity_spare_wake in the slow path influences choice of idlest groups, as we search for groups with maximum spare capacity. In scenarios where RT pressure is high, a sub optimal group can be chosen and hurt performance of the task being woken up.

Several tests with results are included below to show improvements with this change.

1) Hackbench on Pixel 2 Android device (4x4 ARM64 Octa core)

------------------------------------------------------------
Here we have RT activity running on big CPU cluster induced with rt-app, and running hackbench in parallel. The RT tasks are bound to 4 CPUs on the big cluster (cpu 4,5,6,7) and have 100ms periodicity with runtime=20ms sleep=80ms.

Hackbench shows big benefit (30%) improvement when number of tasks is 8 and 32: Note: data is completion time in seconds (lower is better). Number of loops for 8 and 16 tasks is 50000, and for 32 tasks its 20000.

+--------+-----+-------+-------------------+---------------------------+
| groups | fds | tasks | Without Patch     | With Patch                |
+--------+-----+-------+---------+---------+-----------------+---------+
|        |     |       | Mean    | Stdev   | Mean            | Stdev   |
|        |     |       +-------------------+-----------------+---------+
|      1 |   8 |     8 | 1.0534  | 0.13722 | 0.7293 (+30.7%) | 0.02653 |
|      2 |   8 |    16 | 1.6219  | 0.16631 | 1.6391 (-1%)    | 0.24001 |
|      4 |   8 |    32 | 1.2538  | 0.13086 | 1.1080 (+11.6%) | 0.16201 |
+--------+-----+-------+---------+---------+-----------------+---------+

2) Rohit ran barrier.c test (details below) with following improvements:
------------------------------------------------------------------------
This was Rohit's original use case for a patch he posted at [1] however from his recent tests he showed my patch can replace his slow path changes [1] and there's no need to selectively scan/skip CPUs in find_idlest_group_cpu in the slow path to get the improvement he sees.

barrier.c (open_mp code) as a micro-benchmark. It does a number of iterations and barrier sync at the end of each for loop.

Here barrier,c is running in along with ping on CPU 0 and 1 as:
'ping -l 10000 -q -s 10 -f hostX'

barrier.c can be found at:
http://www.spinics.net/lists/kernel/msg2506955.html

Following are the results for the iterations per second with this micro-benchmark (higher is better), on a 44 core, 2 socket 88 Threads Intel x86 machine:

+--------+------------------+---------------------------+
|Threads | Without patch    | With patch                |
|        |                  |                           |
+--------+--------+---------+-----------------+---------+
|        | Mean   | Std Dev | Mean            | Std Dev |
+--------+--------+---------+-----------------+---------+
|1       | 539.36 | 60.16   | 572.54 (+6.15%) | 40.95   |
|2       | 481.01 | 19.32   | 530.64 (+10.32%)| 56.16   |
|4       | 474.78 | 22.28   | 479.46 (+0.99%) | 18.89   |
|8       | 450.06 | 24.91   | 447.82 (-0.50%) | 12.36   |
|16      | 436.99 | 22.57   | 441.88 (+1.12%) | 7.39    |
|32      | 388.28 | 55.59   | 429.4  (+10.59%)| 31.14   |
|64      | 314.62 | 6.33    | 311.81 (-0.89%) | 11.99   |
+--------+--------+---------+-----------------+---------+

3) ping+hackbench test on bare-metal sever (Rohit ran this test)
----------------------------------------------------------------
Here hackbench is running in threaded mode along
with, running ping on CPU 0 and 1 as:
'ping -l 10000 -q -s 10 -f hostX'

This test is running on 2 socket, 20 core and 40 threads Intel x86 machine:
Number of loops is 10000 and runtime is in seconds (Lower is better).

+--------------+-----------------+--------------------------+
|Task Groups   | Without patch   |  With patch              |
|              +-------+---------+----------------+---------+
|(Groups of 40)| Mean  | Std Dev |  Mean          | Std Dev |
+--------------+-------+---------+----------------+---------+
|1             | 0.851 | 0.007   |  0.828 (+2.77%)| 0.032   |
|2             | 1.083 | 0.203   |  1.087 (-0.37%)| 0.246   |
|4             | 1.601 | 0.051   |  1.611 (-0.62%)| 0.055   |
|8             | 2.837 | 0.060   |  2.827 (+0.35%)| 0.031   |
|16            | 5.139 | 0.133   |  5.107 (+0.63%)| 0.085   |
|25            | 7.569 | 0.142   |  7.503 (+0.88%)| 0.143   |
+--------------+-------+---------+----------------+---------+

[1] https://patchwork.kernel.org/patch/9991635/

Matt Fleming also ran cyclictest and several different hackbench tests on his test machines to santiy-check that the patch doesn't harm any of his usecases.

Cc Dietmar Eggemann <[email protected]>

Cc Vincent Guittot <[email protected]>

Cc Morten Ramussen <[email protected]>

Cc Brendan Jackman <[email protected]>

Tested-by Rohit Jain <[email protected]>

Tested-by Matt Fleming <[email protected]>

Signed-off-by Joel Fernandes <[email protected]>

---
 kernel/sched/fair.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

kernel/sched/fair.c +0 -0

index 56f343b8e749..ba9609407cb9 100644

kernel/sched/fair.c +0 -1

 }
 
 /*
--

2.15.0.448.gf294e3d99a-goog

Quoted 5 lines · Joel Fernandes

capacity_spare_wake in the slow path influences choice of idlest groups, as we search for groups with maximum spare capacity. In scenarios where RT pressure is high, a sub optimal group can be chosen and hurt performance of the task being woken up.

Quoted 4 lines

Several tests with results are included below to show improvements with this change.

1) Hackbench on Pixel 2 Android device (4x4 ARM64 Octa core)

"4x4 ARM64 Octa core" is confusing . At least for me, 4x4 means 16 cores :-)

Quoted 10 lines

------------------------------------------------------------
Here we have RT activity running on big CPU cluster induced with rt-app, and running hackbench in parallel. The RT tasks are bound to 4 CPUs on the big cluster (cpu 4,5,6,7) and have 100ms periodicity with runtime=20ms sleep=80ms.

Hackbench shows big benefit (30%) improvement when number of tasks is 8 and 32: Note: data is completion time in seconds (lower is better). Number of loops for 8 and 16 tasks is 50000, and for 32 tasks its 20000.

Quoted 9 lines

+--------+-----+-------+-------------------+---------------------------+
| groups | fds | tasks | Without Patch     | With Patch                |
+--------+-----+-------+---------+---------+-----------------+---------+
|        |     |       | Mean    | Stdev   | Mean            | Stdev   |
|        |     |       +-------------------+-----------------+---------+
|      1 |   8 |     8 | 1.0534  | 0.13722 | 0.7293 (+30.7%) | 0.02653 |
|      2 |   8 |    16 | 1.6219  | 0.16631 | 1.6391 (-1%)    | 0.24001 |
|      4 |   8 |    32 | 1.2538  | 0.13086 | 1.1080 (+11.6%) | 0.16201 |
+--------+-----+-------+---------+---------+-----------------+---------+

Out of curiosity, do you know why you don't see any improvement for
16 tasks but only for 8 and 32 tasks ?

Quoted 9 lines

 kernel/sched/fair.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

kernel/sched/fair.c +0 -0

index 56f343b8e749..ba9609407cb9 100644
@@ -5724,7 +5724,7 @@ static int cpu_util_wake(int cpu, struct task_struct *p);
 

kernel/sched/fair.c static int cpu_util_wake(int cpu, struct task_struct *p); · line 5727

Quoted 4 lines

kernel/sched/fair.c +1 -1

 static unsigned long capacity_spare_wake(int cpu, struct task_struct *p)
 {
-       return capacity_orig_of(cpu) - cpu_util_wake(cpu, p);
+       return max_t(long, capacity_of(cpu) - cpu_util_wake(cpu, p), 0);

Make sense

Reviewed-by Vincent Guittot <[email protected]>

Hi Vincent,

Thanks a lot for your reply, and sorry for the late reply. Actually I just started paternity leave so that's why the delay. My working hours

and completely random at the moment :-)

Quoted 6 lines · Vincent Guittot

Quoted 5 lines · Joel Fernandes

capacity_spare_wake in the slow path influences choice of idlest groups, as we search for groups with maximum spare capacity. In scenarios where RT pressure is high, a sub optimal group can be chosen and hurt performance of the task being woken up.

Quoted 6 lines

Quoted 4 lines

Several tests with results are included below to show improvements with this change.

1) Hackbench on Pixel 2 Android device (4x4 ARM64 Octa core)

"4x4 ARM64 Octa core" is confusing . At least for me, 4x4 means 16 cores :-)

Sure I'll fix it, I meant 4 big and 4 LITTLE CPUs :)

Quoted 20 lines

Quoted 19 lines

------------------------------------------------------------
Here we have RT activity running on big CPU cluster induced with rt-app, and running hackbench in parallel. The RT tasks are bound to 4 CPUs on the big cluster (cpu 4,5,6,7) and have 100ms periodicity with runtime=20ms sleep=80ms.

Hackbench shows big benefit (30%) improvement when number of tasks is 8 and 32: Note: data is completion time in seconds (lower is better). Number of loops for 8 and 16 tasks is 50000, and for 32 tasks its 20000.

+--------+-----+-------+-------------------+---------------------------+
| groups | fds | tasks | Without Patch     | With Patch                |
+--------+-----+-------+---------+---------+-----------------+---------+
|        |     |       | Mean    | Stdev   | Mean            | Stdev   |
|        |     |       +-------------------+-----------------+---------+
|      1 |   8 |     8 | 1.0534  | 0.13722 | 0.7293 (+30.7%) | 0.02653 |
|      2 |   8 |    16 | 1.6219  | 0.16631 | 1.6391 (-1%)    | 0.24001 |
|      4 |   8 |    32 | 1.2538  | 0.13086 | 1.1080 (+11.6%) | 0.16201 |
+--------+-----+-------+---------+---------+-----------------+---------+

Quoted 2 lines

Out of curiosity, do you know why you don't see any improvement for
16 tasks but only for 8 and 32 tasks ?

Yes I'm not fully sure why 16 tasks didn't show that much improvement.

I can try to trace it when I can get a chance. Generally for this test, with more number of tasks, the improvement is lesser. However you're right to point out that the improvement with 32 is > with 16 for this test.

[..]

Quoted 9 lines

Quoted 9 lines

 kernel/sched/fair.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

kernel/sched/fair.c +0 -0

index 56f343b8e749..ba9609407cb9 100644
@@ -5724,7 +5724,7 @@ static int cpu_util_wake(int cpu, struct task_struct *p);
 

Quoted 8 lines

kernel/sched/fair.c static int cpu_util_wake(int cpu, struct task_struct *p); · line 5727

Quoted 4 lines

kernel/sched/fair.c +1 -1

 static unsigned long capacity_spare_wake(int cpu, struct task_struct *p)
 {
-       return capacity_orig_of(cpu) - cpu_util_wake(cpu, p);
+       return max_t(long, capacity_of(cpu) - cpu_util_wake(cpu, p), 0);

Make sense

Reviewed-by Vincent Guittot <[email protected]>

Thanks!

- Joel

Quoted 21 lines

Quoted 21 lines

Quoted 20 lines

------------------------------------------------------------
Here we have RT activity running on big CPU cluster induced with rt-app, and running hackbench in parallel. The RT tasks are bound to 4 CPUs on the big cluster (cpu 4,5,6,7) and have 100ms periodicity with runtime=20ms sleep=80ms.

Hackbench shows big benefit (30%) improvement when number of tasks is 8 and 32: Note: data is completion time in seconds (lower is better). Number of loops for 8 and 16 tasks is 50000, and for 32 tasks its 20000.

+--------+-----+-------+-------------------+---------------------------+
| groups | fds | tasks | Without Patch     | With Patch                |
+--------+-----+-------+---------+---------+-----------------+---------+
|        |     |       | Mean    | Stdev   | Mean            | Stdev   |
|        |     |       +-------------------+-----------------+---------+
|      1 |   8 |     8 | 1.0534  | 0.13722 | 0.7293 (+30.7%) | 0.02653 |
|      2 |   8 |    16 | 1.6219  | 0.16631 | 1.6391 (-1%)    | 0.24001 |
|      4 |   8 |    32 | 1.2538  | 0.13086 | 1.1080 (+11.6%) | 0.16201 |
+--------+-----+-------+---------+---------+-----------------+---------+

Quoted 4 lines

Quoted 2 lines

Out of curiosity, do you know why you don't see any improvement for
16 tasks but only for 8 and 32 tasks ?

Yes I'm not fully sure why 16 tasks didn't show that much improvement.

Yes. This is just to make sure that there no unexpected side effect

Hi Vincent,

Quoted 27 lines

Quoted 21 lines

Quoted 21 lines

Quoted 20 lines

------------------------------------------------------------
Here we have RT activity running on big CPU cluster induced with rt-app, and running hackbench in parallel. The RT tasks are bound to 4 CPUs on the big cluster (cpu 4,5,6,7) and have 100ms periodicity with runtime=20ms sleep=80ms.

Hackbench shows big benefit (30%) improvement when number of tasks is 8 and 32: Note: data is completion time in seconds (lower is better). Number of loops for 8 and 16 tasks is 50000, and for 32 tasks its 20000.

+--------+-----+-------+-------------------+---------------------------+
| groups | fds | tasks | Without Patch     | With Patch                |
+--------+-----+-------+---------+---------+-----------------+---------+
|        |     |       | Mean    | Stdev   | Mean            | Stdev   |
|        |     |       +-------------------+-----------------+---------+
|      1 |   8 |     8 | 1.0534  | 0.13722 | 0.7293 (+30.7%) | 0.02653 |
|      2 |   8 |    16 | 1.6219  | 0.16631 | 1.6391 (-1%)    | 0.24001 |
|      4 |   8 |    32 | 1.2538  | 0.13086 | 1.1080 (+11.6%) | 0.16201 |
+--------+-----+-------+---------+---------+-----------------+---------+

Quoted 4 lines

Quoted 2 lines

Out of curiosity, do you know why you don't see any improvement for
16 tasks but only for 8 and 32 tasks ?

Yes I'm not fully sure why 16 tasks didn't show that much improvement.

Yes. This is just to make sure that there no unexpected side effect

Quoted 30 lines · Joel Fernandes

Hi Vincent,

Quoted 27 lines

Quoted 21 lines

Quoted 21 lines

Quoted 20 lines

------------------------------------------------------------
Here we have RT activity running on big CPU cluster induced with rt-app, and running hackbench in parallel. The RT tasks are bound to 4 CPUs on the big cluster (cpu 4,5,6,7) and have 100ms periodicity with runtime=20ms sleep=80ms.

Hackbench shows big benefit (30%) improvement when number of tasks is 8 and 32: Note: data is completion time in seconds (lower is better). Number of loops for 8 and 16 tasks is 50000, and for 32 tasks its 20000.

+--------+-----+-------+-------------------+---------------------------+
| groups | fds | tasks | Without Patch     | With Patch                |
+--------+-----+-------+---------+---------+-----------------+---------+
|        |     |       | Mean    | Stdev   | Mean            | Stdev   |
|        |     |       +-------------------+-----------------+---------+
|      1 |   8 |     8 | 1.0534  | 0.13722 | 0.7293 (+30.7%) | 0.02653 |
|      2 |   8 |    16 | 1.6219  | 0.16631 | 1.6391 (-1%)    | 0.24001 |
|      4 |   8 |    32 | 1.2538  | 0.13086 | 1.1080 (+11.6%) | 0.16201 |
+--------+-----+-------+---------+---------+-----------------+---------+

Quoted 4 lines

Quoted 2 lines

Out of curiosity, do you know why you don't see any improvement for
16 tasks but only for 8 and 32 tasks ?

Yes I'm not fully sure why 16 tasks didn't show that much improvement.

Yes. This is just to make sure that there no unexpected side effect

It could have been sloppy testing - I could have hit thermal throttling or forgotten to stop Android runtime before running the test. Looking at my old data, the case for 16 tasks has higher completion times than 32 tasks which doesn't make sense. Sorry about that. I was careful this time, I recreated the product tree and applied patch - ran the same test as in this patch, the data prefixed with "with" is with patch and "without" is without patch.

The naming of the Test column is "<test>-<numFDs>-<numGroups>". Data is completion time of hackbench in seconds.

RUN 1:

Test Mean Median Stddev

with-f4-1g  0.67645 (+3.7%)  0.68000 (+3.8%)  0.025755
with-f4-2g  1.0685  (-0.3%)  1.0570 (+1%)       0.044122
with-f4-4g  1.7558  (+0.7%)  1.7685 (+0.08%)    0.096015

without-f4-1g  0.70255  0.70750  0.025330
without-f4-2g  1.0653  1.0680  0.040300
without-f4-4g  1.7688  1.7670  0.046341

RUN 2:

Test         Mean          Median          Stddev
with-f4-1g  0.68100 (+1%)  0.67800 (+2%)   0.025543
with-f4-2g  1.0242 (+1.5%) 1.0260 (+1.5%)  0.042886
with-f4-4g  1.6100 (+3%)   1.6075 (+3.7%)  0.052677

without-f4-1g  0.68840  0.69150  0.030988
without-f4-2g  1.0400  1.0420  0.034288
without-f4-4g  1.6636  1.6670  0.056963

Let me know what you think, thanks.

- Joel

Hi Joel,

Quoted 61 lines · Joel Fernandes

Quoted 30 lines · Joel Fernandes

Hi Vincent,

Quoted 27 lines

Quoted 21 lines

Quoted 21 lines

Quoted 20 lines

------------------------------------------------------------
Here we have RT activity running on big CPU cluster induced with rt-app, and running hackbench in parallel. The RT tasks are bound to 4 CPUs on the big cluster (cpu 4,5,6,7) and have 100ms periodicity with runtime=20ms sleep=80ms.

Hackbench shows big benefit (30%) improvement when number of tasks is 8 and 32: Note: data is completion time in seconds (lower is better). Number of loops for 8 and 16 tasks is 50000, and for 32 tasks its 20000.

+--------+-----+-------+-------------------+---------------------------+
| groups | fds | tasks | Without Patch     | With Patch                |
+--------+-----+-------+---------+---------+-----------------+---------+
|        |     |       | Mean    | Stdev   | Mean            | Stdev   |
|        |     |       +-------------------+-----------------+---------+
|      1 |   8 |     8 | 1.0534  | 0.13722 | 0.7293 (+30.7%) | 0.02653 |
|      2 |   8 |    16 | 1.6219  | 0.16631 | 1.6391 (-1%)    | 0.24001 |
|      4 |   8 |    32 | 1.2538  | 0.13086 | 1.1080 (+11.6%) | 0.16201 |
+--------+-----+-------+---------+---------+-----------------+---------+

Quoted 4 lines

Quoted 2 lines

Out of curiosity, do you know why you don't see any improvement for
16 tasks but only for 8 and 32 tasks ?

Yes I'm not fully sure why 16 tasks didn't show that much improvement.

Yes. This is just to make sure that there no unexpected side effect

It could have been sloppy testing - I could have hit thermal throttling or forgotten to stop Android runtime before running the test. Looking at my old data, the case for 16 tasks has higher completion times than 32 tasks which doesn't make sense. Sorry about that. I was careful this time, I recreated the product tree and applied patch - ran the same test as in this patch, the data prefixed with "with" is with patch and "without" is without patch.

The naming of the Test column is "<test>-<numFDs>-<numGroups>". Data is completion time of hackbench in seconds.

RUN 1:

Test Mean Median Stddev

with-f4-1g  0.67645 (+3.7%)  0.68000 (+3.8%)  0.025755
with-f4-2g  1.0685  (-0.3%)  1.0570 (+1%)       0.044122
with-f4-4g  1.7558  (+0.7%)  1.7685 (+0.08%)    0.096015

without-f4-1g  0.70255  0.70750  0.025330
without-f4-2g  1.0653  1.0680  0.040300
without-f4-4g  1.7688  1.7670  0.046341

RUN 2:

Test         Mean          Median          Stddev
with-f4-1g  0.68100 (+1%)  0.67800 (+2%)   0.025543
with-f4-2g  1.0242 (+1.5%) 1.0260 (+1.5%)  0.042886
with-f4-4g  1.6100 (+3%)   1.6075 (+3.7%)  0.052677

Quoted 6 lines

without-f4-1g  0.68840  0.69150  0.030988
without-f4-2g  1.0400  1.0420  0.034288
without-f4-4g  1.6636  1.6670  0.056963

Let me know what you think, thanks.

The improvement has decreased compared to previous results and there

is instability between your runs; As an example, run2 without patch does better than run1 with patchs for 2g and 4g.

Could you run tests on a SMP linux kernel instead of big/LITTLE android in order to have a saner test environnement and remove some possible disturbances

Hi Vincent,
Thanks for your reply.

Quoted 69 lines · Vincent Guittot

Hi Joel,

Quoted 66 lines · Joel Fernandes

Quoted 30 lines · Joel Fernandes

Hi Vincent,

Quoted 27 lines

Quoted 21 lines

Quoted 21 lines

Quoted 20 lines

------------------------------------------------------------
Here we have RT activity running on big CPU cluster induced with rt-app, and running hackbench in parallel. The RT tasks are bound to 4 CPUs on the big cluster (cpu 4,5,6,7) and have 100ms periodicity with runtime=20ms sleep=80ms.

Hackbench shows big benefit (30%) improvement when number of tasks is 8 and 32: Note: data is completion time in seconds (lower is better). Number of loops for 8 and 16 tasks is 50000, and for 32 tasks its 20000.

+--------+-----+-------+-------------------+---------------------------+
| groups | fds | tasks | Without Patch     | With Patch                |
+--------+-----+-------+---------+---------+-----------------+---------+
|        |     |       | Mean    | Stdev   | Mean            | Stdev   |
|        |     |       +-------------------+-----------------+---------+
|      1 |   8 |     8 | 1.0534  | 0.13722 | 0.7293 (+30.7%) | 0.02653 |
|      2 |   8 |    16 | 1.6219  | 0.16631 | 1.6391 (-1%)    | 0.24001 |
|      4 |   8 |    32 | 1.2538  | 0.13086 | 1.1080 (+11.6%) | 0.16201 |
+--------+-----+-------+---------+---------+-----------------+---------+

Quoted 4 lines

Quoted 2 lines

Out of curiosity, do you know why you don't see any improvement for
16 tasks but only for 8 and 32 tasks ?

Yes I'm not fully sure why 16 tasks didn't show that much improvement.

Yes. This is just to make sure that there no unexpected side effect

It could have been sloppy testing - I could have hit thermal throttling or forgotten to stop Android runtime before running the test. Looking at my old data, the case for 16 tasks has higher completion times than 32 tasks which doesn't make sense. Sorry about that. I was careful this time, I recreated the product tree and applied patch - ran the same test as in this patch, the data prefixed with "with" is with patch and "without" is without patch.

The naming of the Test column is "<test>-<numFDs>-<numGroups>". Data is completion time of hackbench in seconds.

RUN 1:

Test Mean Median Stddev

with-f4-1g  0.67645 (+3.7%)  0.68000 (+3.8%)  0.025755
with-f4-2g  1.0685  (-0.3%)  1.0570 (+1%)       0.044122
with-f4-4g  1.7558  (+0.7%)  1.7685 (+0.08%)    0.096015

without-f4-1g  0.70255  0.70750  0.025330
without-f4-2g  1.0653  1.0680  0.040300
without-f4-4g  1.7688  1.7670  0.046341

RUN 2:

Test         Mean          Median          Stddev
with-f4-1g  0.68100 (+1%)  0.67800 (+2%)   0.025543
with-f4-2g  1.0242 (+1.5%) 1.0260 (+1.5%)  0.042886
with-f4-4g  1.6100 (+3%)   1.6075 (+3.7%)  0.052677

without-f4-1g  0.68840  0.69150  0.030988
without-f4-2g  1.0400  1.0420  0.034288
without-f4-4g  1.6636  1.6670  0.056963

Quoted 3 lines

Quoted 1 line

Let me know what you think, thanks.

The improvement has decreased compared to previous results and there

Yes but the previous result was invalid as I mentioned, I controlled the environment better this time. Previous result showed 4g completed quicker than 2g which wasn't very meaningful.

Quoted 3 lines

is instability between your runs; As an example, run2 without patch does better than run1 with patchs for 2g and 4g.

That's true. The improvement percent isn't stable.

Quoted 3 lines

Could you run tests on a SMP linux kernel instead of big/LITTLE android in order to have a saner test environnement and remove some possible disturbances

Would it be Ok with you if I just dropped this synthetic test from the patch since there are other hackbench results (case 3) from Rohit which are on SMP?

Thanks,

- Joel

kernel/sched/fair.c +0 -0

index 56f343b8e749..ba9609407cb9 100644