Scheduler behaviour
We discovered performance degradation with dbench when using kernel 2.6.23 compared to kernel 2.6.22.
In our case we booted a Linux in a IBM System z9 LPAR with 256MB of ram with 4 CPU's. This system uses a striped LV with 16 disks on a Storage Server connected via 8 4GBit links.
A dbench was started on that system performing I/O operations on the striped LV. dbench runs were performed with 1 to 62 processes. Measurements with a 2.6.22 kernel were compared to measurements with a 2.6.23 kernel. We saw a throughput degradation from 7.2 to 23.4 percent and a cost increase from 9.5 to 29.5 percent. Costs are calculated by consumed CPU microseconds divided by transferred bytes written/read.
The cost increase is caused by fewer transferred bytes at an almost constant level of spent CPU microseconds (except for 50 processes).
The throughput can be increased by generating an imbalance in scheduling of the dbench processes with different nice value for the processes. The more imbalance is created the higher is the throughput. By monitoring the throughput of disk I/O it can be observed that the iostat read throughput is significantly lower with this imbalance while the iostat write throughput stays the same.
|-----------------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------> | | Device:| rrqm/s| wrqm/s| r/s| w/s| rsec/s| wsec/s| avgrq-sz| avgqu-sz| await| |-----------------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+----------->
Quoted 1 line
-----------+-----------|
| svctm| %util| >-----------+-----------| |-----------------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------> |Balanced | dm-0| 0| 0| 10592.54| 8067.16| 694885.57| 510479.6| 64.6| 16.96| 0.91| |scheduling | | | | | | | | | | | |-----------------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------> >-----------+-----------| | 0.05| 97.51| >-----------+-----------| |-----------------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------> |Imbalanced | dm-0| 0| 0| 3401.00| 7993.03| 175693.53| 526833.83| 61.66| 14.88| 1.3| |scheduling | | | | | | | | | | | |-----------------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------> >-----------+-----------| | 0.07| 83.58| >-----------+-----------|
With profiling we saw that the mpage_end_io_read function consumes much more CPU with a 2.6.23 kernel than with a 2.6.22 kernel.
To me it looks like the page cache is under stronger pressure when all processes are scheduled fairly.
Normalized Throughput of dbench
Number of 2.6.22 2.6.23 Difference
Processes in Percent
1 1 0.9 -10.24%
4 3.64 3.74 2.68%
8 3.57 3.71 3.83%
12 3.51 3.6 2.55%
16 3.4 3.53 3.96%
20 3.32 3.43 3.33%
26 3.29 3.4 3.29%
32 3.14 2.92 -7.25%
40 2.99 2.61 -12.92%
46 3 2.47 -17.69%
50 2.84 2.4 -15.55%
54 3.1 2.37 -23.42%
62 2.55 2.32 -8.99% Normalized Costs of dbench
Number of 2.6.22 2.6.23 Difference
Processes in Percent
1 1 0.96 3.84%
4 1.08 1.05 2.97%
8 1.08 1.07 1.10%
12 1.09 1.1 -1.30%
16 1.1 1.12 -1.90%
20 1.12 1.15 -2.61%
26 1.18 1.17 0.86%
32 1.24 1.36 -9.50%
40 1.3 1.52 -16.97%
46 1.29 1.6 -23.92%
50 1.37 1.64 -20.12%
54 1.28 1.66 -29.60%
62 1.5 1.69 -12.84% regards
Holger Wolf
Quoted 1 line · Holger Wolf
We discovered performance degradation with dbench when using kernel 2.6.23 compared to kernel 2.6.22.
We've fixed a lot of regressions and made a lot of other changes to the scheduler since .23. Could you please try the backport of the latest scheduler on top of .23 to see if any of our recent work solved your problem? (using a backport against .23 or even .22 allows you to test just the scheduler changes in isolation)
http://people.redhat.com/mingo/cfs-scheduler/sched-cfs-v2.6.22.13-v24.patch
http://people.redhat.com/mingo/cfs-scheduler/sched-cfs-v2.6.23.9-v24.patch
Also, it could be dbench is one of those benchmarks that favours unfairness, I seem to remember so.. still we should not regress too much without a very good reason.
Quoted 8 lines · Peter Zijlstra
Quoted 2 lines
We discovered performance degradation with dbench when using kernel 2.6.23 compared to kernel 2.6.22.
We've fixed a lot of regressions and made a lot of other changes to the scheduler since .23. Could you please try the backport of the latest scheduler on top of .23 to see if any of our recent work solved your problem? (using a backport against .23 or even .22 allows you to test just the scheduler changes in isolation)
actually, i'd suggest to use the CFS backport against .22 right away:
that would exclude all other changes that happened in 2.6.23.
Ingo
Quoted 10 lines · On Wed, 05 Dec 2007 21:15:30 +0100 Holger Wolf <[email protected]> wrote:
We discovered performance degradation with dbench when using kernel 2.6.23 compared to kernel 2.6.22.
In our case we booted a Linux in a IBM System z9 LPAR with 256MB of ram with 4 CPU's. This system uses a striped LV with 16 disks on a Storage Server connected via 8 4GBit links.
A dbench was started on that system performing I/O operations on the striped LV. dbench runs were performed with 1 to 62 processes. Measurements with a 2.6.22 kernel were compared to measurements with a 2.6.23 kernel. We saw a throughput degradation from 7.2 to 23.4
this is good news!
dbench rewards unfair behavior... so higher dbench usually means a worse kernel ;)
Signature · 3 lines
If you want to reach me at my work email, use [email protected] For development, discussion and tips for power savings, visit http://www.lesswatts.org
Arjan van de Ven wrote, On 12/05/2007 10:26 PM:
Quoted 2 lines
On Wed, 05 Dec 2007 21:15:30 +0100
Holger Wolf <[email protected]> wrote:
...
Quoted 5 lines
Quoted 1 line
a 2.6.23 kernel. We saw a throughput degradation from 7.2 to 23.4
this is good news!
dbench rewards unfair behavior... so higher dbench usually means a worse kernel ;)
Yes!!!
(But, with those new patches, I guess, our kernel will be worse enough?!)
Jarek P.
Quoted 16 lines · Arjan van de Ven
Quoted 11 lines · On Wed, 05 Dec 2007 21:15:30 +0100 Holger Wolf <[email protected]> wrote:
We discovered performance degradation with dbench when using kernel 2.6.23 compared to kernel 2.6.22.
In our case we booted a Linux in a IBM System z9 LPAR with 256MB of ram with 4 CPU's. This system uses a striped LV with 16 disks on a Storage Server connected via 8 4GBit links.
A dbench was started on that system performing I/O operations on the striped LV. dbench runs were performed with 1 to 62 processes. Measurements with a 2.6.22 kernel were compared to measurements with a 2.6.23 kernel. We saw a throughput degradation from 7.2 to 23.4
Quoted 6 lines
this is good news!
dbench rewards unfair behavior... so higher dbench usually means a worse kernel ;)
tests with 2.6.22 including CFS show the same results.
This means the pressure on page cache is much higher when all processes run in parallel.
We see this behavior as well with iozone when writing on many disks with many threads and just 256 MB memory.
This means the scheduler schedules as it should - fair.
regards Holger