reaim - 2.6.3-mm1 IO performance down.
Running the reaim 'new_fserver' workload, we now see a performance drop on 2.6.3-mm1, ext3 filesystem
Kernel JPM Max Users Percent change inux-2.6.3 10347.87 172 0.0 2.6.3-rc3-mm1 9826.35 164 -5.07 2.6.3-mm1 8938.17 140 (run 1) -13.65 2.6.3-mm1 9100.39 136 (run 2)-12.08
I have done some comparions graphs here, with readprofile data:
http://developer.osdl.org/cliffw/reaim/compares/r_comp/2.6.3_vs_mm1_r1/index.html
http://developer.osdl.org/cliffw/reaim/compares/r_comp/2.6.3_vs_mm1_r2/index.html
The interesting graph is the bottom one on the page, comparing client run-time to number of children. The -mm1 kernel has a spike in run time, becoming very noticealbe around 60-80 lusers.
kernel Users JPM Run Time linux-2.6.3 60 10327.87 34.16 seconds 2.6.3-mm1 60 8279.75 42.61 seconds
Linux-2.6.3 80 10275.23 45.78 seconds 2.6.3-mm1 80 7841.31 59.99 seconds
For the same test on the same machine, results from 2.6.2-rc1-mm2 and 2.6.2-rc3-mm1 were within 1.0% of the linux-2.6.2 runs. So this is new.
More data and tests if requested - are there some patch sets we should try reverting? cliffw
Signature · 2 lines
The church is near, but the road is icy. The bar is far, but i will walk carefully. - Russian proverb
Quoted 5 lines · cliff white
For the same test on the same machine, results from 2.6.2-rc1-mm2 and 2.6.2-rc3-mm1 were within 1.0% of the linux-2.6.2 runs. So this is new.
More data and tests if requested - are there some patch sets we should try reverting?
Thanks. You could try reverting adaptive-lazy-readahead.patch. If it is not that I'd be suspecting CPU scheduler changes. Do you have uniprocessor test results?
Quoted 4 lines
Quoted 2 lines · cliff white
For the same test on the same machine, results from 2.6.2-rc1-mm2 and 2.6.2
-rc3-mm1
Quoted 8 lines
Quoted 3 lines
were within 1.0% of the linux-2.6.2 runs. So this is new.
More data and tests if requested - are there some patch sets we should try
reverting?
Thanks. You could try reverting adaptive-lazy-readahead.patch. If it is not that I'd be suspecting CPU scheduler changes. Do you have uniprocessor test results?
I have them for 2.6.3-mm3, am re-running 2.6.3-mm1 right now. Gross results are within 1%, but looking at the detail, i do see badness, example:
Kernel Users Run time
2.6.3 20 32.11
2.6.3-mm3 20 35.47
2.6.3 40 63.64
2.6.3-mm2 40 66.33
Again, this shows up best on the bottom graph on the page. Graphn of 2.6.3 vs 2.6.3-mm3 :
http://developer.osdl.org/cliffw/reaim/compares/r_comp/2.6.3_vs_mm1_1cpu/index.html
cliffw
Quoted 8 lines
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Quoted 7 lines · Cliff White
Quoted 4 lines
Thanks. You could try reverting adaptive-lazy-readahead.patch. If it is not that I'd be suspecting CPU scheduler changes. Do you have uniprocessor test results?
I have them for 2.6.3-mm3, am re-running 2.6.3-mm1 right now. Gross results are within 1%, but looking at the detail, i do see badness,
OK, I'll do the bsearch. Again. Could you please tell me what would be an appropriate reaim command line with which to reproduce this?
Quoted 5 lines
Quoted 2 lines · Cliff White
Quoted 2 lines
Thanks. You could try reverting adaptive-lazy-readahead.patch. If it i
s
Quoted 1 line
Quoted 1 line
not that I'd be suspecting CPU scheduler changes. Do you have uniproces
Quoted 8 lines
sor
Quoted 4 lines
Quoted 1 line
test results?
I have them for 2.6.3-mm3, am re-running 2.6.3-mm1 right now. Gross results are within 1%, but looking at the detail, i do see badness,
OK, I'll do the bsearch. Again. Could you please tell me what would be an appropriate reaim command line with which to reproduce this?
Sure.
We use the 'workfile.new_fserver' for this load. Comes in the kit. This test does IO, so you'll need to list some directories and a filepool size
in a text file. Ours looks like this:
stp.config
-------------------
FILESIZE 80k
POOLSIZE 1024k
DISKDIR /mnt/disk1
DISKDIR /mnt/disk2
DISKDIR /mnt/disk3
DISKDIR /mnt/disk4
--------------------
If you want the shortest possible run, shrink POOLSIZE and go with the defaults:
reaim -fworkfile.new_fserver -l./stp.config
That will run 1->5 users, should be < 1 egg timer in duration on modern iron. I dunno if that'll be a useful data point. To replicate at a specific number of users, pick a number ( example: 20 )
, and use this:
reaim -s20 -e20 -fworkfile.new_fserver -l./stp.config
That will do one pass at 20 users. Should be quick, maybe 2-4 egg timers, and will give you one of the data points I reported. If you have more time, this runs range 20->80 with a 10 user increment:
reaim -s20 -e80 -i10 -t -fworkfile.new_fserver -l./stp.config ( the -t turns off the AIM7 adaptive increment )
The STP runs are very long. Many egg-timers.
We invoke them like this, where NCPU is the number of CPUS we have:
The 'quick convergence run' ( repeats three times )
reaim -s$NCPU -q -t -i$NCPU -fworkfile.new_fserver -r3 -b -l./stp.config
This is the 'maximum' longest run
reaim -s$NCPU -s -t -i$NCPU -fworkfile.new_fserver -r3 -b -l./stp.config
If you want those, quicker to request via STP/PLM - cliffw
Quoted 7 lines
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Quoted 4 lines
Quoted 2 lines · cliff white
For the same test on the same machine, results from 2.6.2-rc1-mm2 and 2.6.2
-rc3-mm1
Quoted 8 lines
Quoted 3 lines
were within 1.0% of the linux-2.6.2 runs. So this is new.
More data and tests if requested - are there some patch sets we should try
reverting?
Thanks. You could try reverting adaptive-lazy-readahead.patch. If it is not that I'd be suspecting CPU scheduler changes. Do you have uniprocessor test results?
adaptive-lazy-readahead reverted, not really much change here:
Kernel Users JPM Run Time
2.6.3 60 10327.87 34.16 seconds
2.6.3-mm1 60 8279.75 42.61 seconds
2.6.3-mm1-noalr 60 7731.76 45.63 seconds
2.6.3 80 10275.23 45.78 seconds
2.6.3-mm1 80 7841.31 59.99 seconds
2.6.3-mm1-noalr 80 8565.19 54.92 seconds
Full details, reaim tarball:
http://developer.osdl.org/cliffw
cliffw
Quoted 8 lines
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/