PROBLEM: Consistently slower 3ware RAID performance under 2.6.4
## Background:
I'm doing a lot of sequential read and write performance tests on my 3ware 7000-2 controller (RAID1 with 2xWD1200JB), because I'm getting very poor write performance. This is a problem that I'm currently working with 3ware to resolve, and they are working enthusiastically with me to fix it.
Read test: time dd if=/dev/sda2 of=/dev/null bs=1024k count=1024 Write test: time dd if=/dev/zero of=/dev/sda2 bs=1024k count=1024
NOTE: /dev/sda2 is the swap partition which is NEAR the outer-most tracks.
## The new problem I discovered specific to Linux:
Regardless of the above-mentioned problem, I am noticing a very significant performance drop between a 2.4 kernel and a 2.6 kernel.
## Performance with "2.4.25-gentoo":
The read test here takes 21.6 seconds which is about 47MB/sec. This is a correct number, because I have measured the maximum read throughput from each drive to be 47MB/sec.
The write test here takes 2 minutes, 2.5 seconds. This translates to 8.35MB/sec. This is what I'm working with 3ware to correct, but let's call this the baseline write performance.
## Performance with "2.6.4-gentoo-r1":
The read test here takes 33.9 seconds. That's down to about 30MB/sec.
The write test here takes 2 minutes, 44.2 seconds. That is down to 6.2MB/sec.
## HELP?
How can I help kernel developers to diagnose this problem? What information do I need to provide that is missing?
What is responsible for such a significant performance drop on LONG sequential disk accesses?
## My computer:
Athlon 2800+
1GB RAM (Corsair 2700LL)
ABIT KD7 (KT400 chipset)
Thanks in advance for the help!
Quoted 12 lines · Timothy Miller
## Performance with "2.4.25-gentoo":
The read test here takes 21.6 seconds which is about 47MB/sec. This is a correct number, because I have measured the maximum read throughput from each drive to be 47MB/sec.
The write test here takes 2 minutes, 2.5 seconds. This translates to 8.35MB/sec. This is what I'm working with 3ware to correct, but let's call this the baseline write performance.
Quoted 7 lines
## Performance with "2.6.4-gentoo-r1":
The read test here takes 33.9 seconds. That's down to about 30MB/sec.
The write test here takes 2 minutes, 44.2 seconds. That is down to 6.2MB/sec.
You cannot compare 2.4 and 2.6 kernel write performance with `dd', because the kernels are tuned differently. 2.6 kernels are tuned to leave less dirty pages in memory than a 2.4 kernel. Hence when your dd has finished, 40% of memory will be dirty (needing writeout) under 2.6, but this figure is 60% on 2.4.
So the 2.6 kernel does more writeout during the dd run and less writeout after dd has finished. The 2.4 kernel does less writeout during the dd run and more writeout after dd has finished.
To compare IO performance you'll need to set 2.6's /proc/sys/vm/dirty_ratio to 60 and /proc/sys/vm/dirty_async_ratio to 30. Or use write-and-fsync from http://www.zip.com.au/~akpm/linux/patches/stuff/ext3-tools.tar.gz with the `-f' option.
I don't know why the read bandwidth appears to be lower. Try increasing the readahead with `blockdev --setra'?
Quoted 11 lines · Andrew Morton
You cannot compare 2.4 and 2.6 kernel write performance with `dd', because the kernels are tuned differently. 2.6 kernels are tuned to leave less dirty pages in memory than a 2.4 kernel. Hence when your dd has finished, 40% of memory will be dirty (needing writeout) under 2.6, but this figure is 60% on 2.4.
So the 2.6 kernel does more writeout during the dd run and less writeout after dd has finished. The 2.4 kernel does less writeout during the dd run and more writeout after dd has finished.
Quoted 7 lines
To compare IO performance you'll need to set 2.6's
/proc/sys/vm/dirty_ratioto 60 and/proc/sys/vm/dirty_async_ratioto 30. Or use write-and-fsync from http://www.zip.com.au/~akpm/linux/patches/stuff/ext3-tools.tar.gz with the `-f' option.I don't know why the read bandwidth appears to be lower. Try increasing the readahead with `blockdev --setra'?
That's odd. I've tried different mem= kernel parameters with no change in throughput under 2.4. That is, 1G vs. 128M vs. 32M all perform the same under 2.4 when doing dd to a block device. On the other hand, available RAM makes a HUGE difference when doing dd to a _file_system_.
I haven't tried that under 2.6 yet.
Anyhow, I chose a 1GB test run so as diminish cache effects and file system overhead. Is my logic flawed here?
Thanks.