Skip to contentSkip to conversation

reaim - 2.6.3-mm1 IO performance down.

DISCUSSION cliff white 2 participants 6 messages Feb 25, 2004

lore
Share & tools
lore
#0 lore raw

Running the reaim 'new_fserver' workload, we now see a performance drop on 2.6.3-mm1, ext3 filesystem

Kernel          JPM       Max Users     Percent change
inux-2.6.3      10347.87  172           0.0
2.6.3-rc3-mm1   9826.35   164           -5.07
2.6.3-mm1       8938.17   140   (run 1) -13.65
2.6.3-mm1       9100.39   136    (run 2)-12.08 

I have done some comparions graphs here, with readprofile data:
http://developer.osdl.org/cliffw/reaim/compares/r_comp/2.6.3_vs_mm1_r1/index.html
http://developer.osdl.org/cliffw/reaim/compares/r_comp/2.6.3_vs_mm1_r2/index.html

The interesting graph is the bottom one on the page, comparing client run-time to number of children. The -mm1 kernel has a spike in run time, becoming very noticealbe around 60-80 lusers.

kernel       Users 	JPM        Run Time
linux-2.6.3  60         10327.87   34.16 seconds
2.6.3-mm1    60         8279.75    42.61 seconds
Linux-2.6.3  80         10275.23   45.78 seconds
2.6.3-mm1    80         7841.31    59.99 seconds

For the same test on the same machine, results from 2.6.2-rc1-mm2 and 2.6.2-rc3-mm1 were within 1.0% of the linux-2.6.2 runs. So this is new.

More data and tests if requested - are there some patch sets we should try reverting? cliffw

Signature · 2 lines
The church is near, but the road is icy.
The bar is far, but i will walk carefully. - Russian proverb
#1 lore raw
Quoted 5 lines · cliff white

For the same test on the same machine, results from 2.6.2-rc1-mm2 and 2.6.2-rc3-mm1 were within 1.0% of the linux-2.6.2 runs. So this is new.

More data and tests if requested - are there some patch sets we should try reverting?

Thanks. You could try reverting adaptive-lazy-readahead.patch. If it is not that I'd be suspecting CPU scheduler changes. Do you have uniprocessor test results?

#2 lore raw
Quoted 4 lines
Quoted 2 lines · cliff white

For the same test on the same machine, results from 2.6.2-rc1-mm2 and 2.6.2

-rc3-mm1

Quoted 8 lines
Quoted 3 lines

were within 1.0% of the linux-2.6.2 runs. So this is new.

More data and tests if requested - are there some patch sets we should try

reverting?

Thanks. You could try reverting adaptive-lazy-readahead.patch. If it is not that I'd be suspecting CPU scheduler changes. Do you have uniprocessor test results?

I have them for 2.6.3-mm3, am re-running 2.6.3-mm1 right now. Gross results are within 1%, but looking at the detail, i do see badness, example:

Kernel Users Run time
2.6.3 20 32.11
2.6.3-mm3 20 35.47

2.6.3 40 63.64
2.6.3-mm2 40 66.33

Again, this shows up best on the bottom graph on the page. Graphn of 2.6.3 vs 2.6.3-mm3 :
http://developer.osdl.org/cliffw/reaim/compares/r_comp/2.6.3_vs_mm1_1cpu/index.html
cliffw

Quoted 8 lines

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#3 lore raw
Quoted 7 lines · Cliff White
Quoted 4 lines

Thanks. You could try reverting adaptive-lazy-readahead.patch. If it is not that I'd be suspecting CPU scheduler changes. Do you have uniprocessor test results?

I have them for 2.6.3-mm3, am re-running 2.6.3-mm1 right now. Gross results are within 1%, but looking at the detail, i do see badness,

OK, I'll do the bsearch. Again. Could you please tell me what would be an appropriate reaim command line with which to reproduce this?

#4 lore raw
Quoted 5 lines
Quoted 2 lines · Cliff White
Quoted 2 lines

Thanks. You could try reverting adaptive-lazy-readahead.patch. If it i

s

Quoted 1 line
Quoted 1 line

not that I'd be suspecting CPU scheduler changes. Do you have uniproces

Quoted 8 lines

sor

Quoted 4 lines
Quoted 1 line

test results?

I have them for 2.6.3-mm3, am re-running 2.6.3-mm1 right now. Gross results are within 1%, but looking at the detail, i do see badness,

OK, I'll do the bsearch. Again. Could you please tell me what would be an appropriate reaim command line with which to reproduce this?

Sure.
We use the 'workfile.new_fserver' for this load. Comes in the kit. This test does IO, so you'll need to list some directories and a filepool size
in a text file. Ours looks like this:
stp.config
-------------------
FILESIZE 80k
POOLSIZE 1024k
DISKDIR /mnt/disk1
DISKDIR /mnt/disk2
DISKDIR /mnt/disk3
DISKDIR /mnt/disk4
--------------------

If you want the shortest possible run, shrink POOLSIZE and go with the defaults:
reaim -fworkfile.new_fserver -l./stp.config

That will run 1->5 users, should be < 1 egg timer in duration on modern iron. I dunno if that'll be a useful data point. To replicate at a specific number of users, pick a number ( example: 20 )
, and use this:

reaim -s20 -e20 -fworkfile.new_fserver -l./stp.config

That will do one pass at 20 users. Should be quick, maybe 2-4 egg timers, and will give you one of the data points I reported. If you have more time, this runs range 20->80 with a 10 user increment:

reaim -s20 -e80 -i10 -t -fworkfile.new_fserver -l./stp.config ( the -t turns off the AIM7 adaptive increment )

The STP runs are very long. Many egg-timers.
We invoke them like this, where NCPU is the number of CPUS we have:
The 'quick convergence run' ( repeats three times )
reaim -s$NCPU -q -t -i$NCPU -fworkfile.new_fserver -r3 -b -l./stp.config

This is the 'maximum' longest run
reaim -s$NCPU -s -t -i$NCPU -fworkfile.new_fserver -r3 -b -l./stp.config

If you want those, quicker to request via STP/PLM - cliffw

Quoted 7 lines

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#5 lore raw
In reply to Andrew Morton
Quoted 4 lines
Quoted 2 lines · cliff white

For the same test on the same machine, results from 2.6.2-rc1-mm2 and 2.6.2

-rc3-mm1

Quoted 8 lines
Quoted 3 lines

were within 1.0% of the linux-2.6.2 runs. So this is new.

More data and tests if requested - are there some patch sets we should try

reverting?

Thanks. You could try reverting adaptive-lazy-readahead.patch. If it is not that I'd be suspecting CPU scheduler changes. Do you have uniprocessor test results?

adaptive-lazy-readahead reverted, not really much change here:
Kernel Users JPM Run Time
2.6.3 60 10327.87 34.16 seconds
2.6.3-mm1 60 8279.75 42.61 seconds
2.6.3-mm1-noalr 60 7731.76 45.63 seconds

2.6.3 80 10275.23 45.78 seconds
2.6.3-mm1 80 7841.31 59.99 seconds
2.6.3-mm1-noalr 80 8565.19 54.92 seconds

Full details, reaim tarball:
http://developer.osdl.org/cliffw

cliffw

Quoted 8 lines

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/