gdth / SCSI read performance issues (2.2.19 and 2.4.10)
Hi,
our brand-new ICP GDT8523RZ controller with 6 disks peaks out at 45MB/s under 2.4.10, while with 2.2.19 it reaches 85MB/s (seq. read performance). It should realistically be able to reach at least 150-200MB/s in this configuration (RAID-5, 6 disks, it's a 64-bit 66MHz PCI card). It's in a dual P3-1GHz box with Tyan 2510NG board. Sequential write performance is OK with both kernels, it's higher than the read performance on 2.4.10 (60MB/s).
Under 2.4.10, when I test the performance with a simple program that just read()'s 16MB blocks from /dev/sda, both CPUs report 40-50% System time usage. 2.2.19 reports ~20% for one CPU (and better performance).
Is there anything that can be done about this? It seems like a serious performance problem either with the gdth driver or the kernel and it renders this neat controller rather ineffective.
Regards,
-mjy
[please CC: answers]
Quoted 7 lines · Marinos Yannikos
Hi,
our brand-new ICP GDT8523RZ controller with 6 disks peaks out at 45MB/s under 2.4.10, while with 2.2.19 it reaches 85MB/s (seq. read performance). It should realistically be able to reach at least 150-200MB/s in this configuration
Well that's pretty bad, isn't it?
Some things which it would be interesting to try out:
- Disable CONFIG_HIGHMEM in your kernel config, see what that does.
- Try linux-2.4.13 -- 2.4.10 was a bit of a dog (this may make a difference if the driver uese the new PCI DMA API, but it doesn't explain why 2.2.x does so much better).
- Profile the kernel. You may enable profiling by booting the kernel with the LILO option `profile=1'.
Then, making sure that /boot/System.map reflects the running kernel, run this little script
#!/bin/sh
TIMEFILE=/tmp/$(basename $1).time
sudo readprofile -r
time "$@"
readprofile -v -m /boot/System.map | sort -n +2 | tail -40 | tee $TIMEFILE echo created $TIMEFILE
With something like:
~/kern-prof.sh cp some_huge_file /dev/null
Quoted 1 line · Andrew Morton
Well that's pretty bad, isn't it?
We could have bought a much cheaper controller and slower disks ;-)
Quoted 1 line
- Disable
CONFIG_HIGHMEM
That seems to have no effect on performance. Btw., the 64GB support in 2.4.10 seemed to be buggy ("0-order allocation failed", then the DB crashed), so we were using the 4GB setting.
Quoted 1 line
- Try linux-2.4.13
This helped - now performance is up to par with 2.2.19 (~ 85MB/s) - thanks!
Quoted 2 lines
- Profile the kernel. [...] With something like:
~/kern-prof.shcpsome_huge_file/dev/null
I tried this, but with
dd if=/dev/sda of=/dev/null bs=1024k count=3000
(the fs is too slow - so "cp" peaks out at 17MB/s!)
The result (last 4 lines):c01388fc try_to_free_buffers 55 0.1511
c0128b10 file_read_actor 1179 14.0357 c01053b0 default_idle 6784 130.4615 00000000 total 8695 0.0065
Does this suggest that the kernel isn't the bottleneck?
Regards,
Marinos
Signature · 4 lines
Marinos Yannikos, CEO Preisvergleich Internet Services AG Franzensbrückenstraße 8/2/16, A-1020 Wien Tel./Fax: (+431) 5811609-52/-55
Quoted 6 lines · Marinos J. Yannikos
Quoted 1 line · Andrew Morton
Well that's pretty bad, isn't it?
We could have bought a much cheaper controller and slower disks ;-)
It's Linux's way of saving you money :)
Quoted 5 lines
Quoted 1 line
- Disable
CONFIG_HIGHMEMThat seems to have no effect on performance. Btw., the 64GB support in 2.4.10 seemed to be buggy ("0-order allocation failed", then the DB crashed), so we were using the 4GB setting.
There have been many reports of this happening, but they are tapering off now.
Quoted 3 lines
Quoted 1 line
- Try linux-2.4.13
This helped - now performance is up to par with 2.2.19 (~ 85MB/s) - thanks!
It's surprising that 2.4.10->2.4.13 actually doubled throughput.
Quoted 7 lines
Quoted 2 lines
- Profile the kernel. [...] With something like:
~/kern-prof.shcpsome_huge_file/dev/nullI tried this, but with
dd if=/dev/sda of=/dev/null bs=1024k count=3000(the fs is too slow - so "cp" peaks out at 17MB/s!)
Really? Are you saying that on a controller which can do 85 megs/second, you can't read files through the filesystem at greater than 17? Which filesystem?
Quoted 5 lines
The result (last 4 lines):
c01388fctry_to_free_buffers55 0.1511c0128b10 file_read_actor 1179 14.0357 c01053b0 default_idle 6784 130.4615 00000000 total 8695 0.0065
OK, that's normal and proper. Almost all the kernel time is spent copying data.
Quoted 1 line
Does this suggest that the kernel isn't the bottleneck?
Well... We seem to have three issues here:
1: Why isn't the controller achieving the manufacturer's
claimed throughput?
Don't know. Maybe it's the software copy. Maybe it's the device driver. Maybe they lied :). It'd be interesting to test it on the same machine with the vendor's drivers and win2k.
2: 0-order allocation failures.
3: Poor `cp' throughput. This one is strange. Perhaps
`cp' is using a small transfer-size and the kernel's readhead isn't working properly. Could you experiment with this some more? For example, what happens with
dd if=large_file of=/dev/null bs=4096k
-
Quoted 2 lines · Andrew Morton
[...] you can't read files through the filesystem
at greater than 17? Which filesystem?
ext2 (on which I did a "tune2fs -j", but then I found out that the kernel apparently doesn't support ext3). But, see below.
Quoted 3 lines
[...] It'd be interesting to test it on the same machine with
the vendor's drivers and win2k.
Indeed - can't do so, unfortunately. It could also be an issue with the RAID-5 configuration (block size and arrangement of the disks in the array could be less than optimal as suggested by the ICP BIOS), so I'm in touch with the very helpful ICP support people as well.
Quoted 1 line
dd if=large_file of=/dev/null bs=4096k
OK, that's quite odd - now I get a reasonable ~55 MB/s whatever I try (dd, cp), so that must have been my mistake (or due to the kernel profiling option?).
By the way, I'm having lock-ups with 2.4.13 that I can reproduce (vi <file on reiserfs partition> ... :q ... "Segmentation fault" ... lock-up - console input still works, but apparently processes don't get scheduled).
Regards,
Marinos
Signature · 4 lines
Marinos Yannikos, CEO Preisvergleich Internet Services AG Franzensbrückenstraße 8/2/16, A-1020 Wien Tel./Fax: (+431) 5811609-52/-55