Linux Page Cache performance

Hi All.

We are running dbench on a machine with Dual Xeon (Hyper threading turned off), 1GB RAM
and 4 Drive software RAID5. The kernel is 2.4.29-xfs 1.2. We are using LVM. However
similar test done using ext2 on a disk partiotion (no md or LVM) shows

The throughput is find till the number of clients are Around 16. At that point the throughput
plummets about 40%. We are trying to avoid that and see how we could have a consistent throughput
perhaps sacrificing some peak performance.

One could argue that at the point the performance drops we are actualy beggining to see
I/O requests missing page cache and going to disk. That is our current guess. We try to tune
the buffer age and the interval page buffer daemon runs, but had no effect on the curve.
So transactional meta data operations seems not be causing the bottle neck.

Are there any VM patches that smooths out the Page Cache dirty page swapping process so that we
wont see this sudden drop of through put, but could have a smoother transition. I ran a
kernel profiler on the test and I dont see any dentry cache flushes or inode cache flushes.

Similar test done using ext2 on a disk partiotion (no md or LVM) shows us similar performances.
However for ext2 when we tweak the bdflush parameters in /proc (specifically the max age of meta
data buffers) we can push the point where the data throughput falls to around 40 clients.

But we are more interested in finding out ways to solve the XFS case.

Any insights on this ?

Thanx
Ravi

=====
------------------------------
Ravi Wijayaratne

__________________________________
Do you Yahoo!?
Yahoo! Hotjobs: Enter the "Signing Bonus" Sweepstakes
http://hotjobs.sweepstakes.yahoo.com/signingbonus

First off, dbench is a bad benchmark to tune for, it shows better numbers for worse fairness.

Is yourworkload like dbench? What will your workload be?

Of course, there's 2.4.24 which has XFS merged for you, and be sure to give 2.4.25-pre6 a try since it integrates part of the -aa VM for inodes in highmem.

There are options in XFS that change the layout of the filesystem that might help (can't say more since I'm not too familiar with XFS).

Try without highmem, and see if that changes your results, and try with a benchmark that behaves similair to how you expect from your workload.

Also, there's 2.6 which will give different characteristics, and especially with the disk IO schedulers available there.

Once the amount of dirty memory in the machine hits 40% of the total, the VM starts to initiate writeout. At 60% dirty the VM starts to deliberately throttle the processes which are dirtying memory.

So you would expect to see extremely sudden transitions in overall throughput at that point. Note that if the test were to run for a longer period of time, or if it were to leave the files behind rather than suddenly removing them you would not notice this effect.

The probability that dbench's access patterns have any similarity to the access patterns of the application which you care about is vanishingly small. The same can most probably be said of all the other off-the-shelf benchmarks. They're not very impressive.

You will need to analyse your application's access patterns and design a benchmark which reasonably closely and repeatably models them. Or, if possible, use the live application itself.