Skip to contentSkip to conversation

gdth / SCSI read performance issues (2.2.19 and 2.4.10)

DISCUSSION Marinos Yannikos 2 participants 5 messages Oct 24, 2001

lore
Share & tools
lore
#0 lore raw

Hi,

our brand-new ICP GDT8523RZ controller with 6 disks peaks out at 45MB/s under 2.4.10, while with 2.2.19 it reaches 85MB/s (seq. read performance). It should realistically be able to reach at least 150-200MB/s in this configuration (RAID-5, 6 disks, it's a 64-bit 66MHz PCI card). It's in a dual P3-1GHz box with Tyan 2510NG board. Sequential write performance is OK with both kernels, it's higher than the read performance on 2.4.10 (60MB/s).

Under 2.4.10, when I test the performance with a simple program that just read()'s 16MB blocks from /dev/sda, both CPUs report 40-50% System time usage. 2.2.19 reports ~20% for one CPU (and better performance).

Is there anything that can be done about this? It seems like a serious performance problem either with the gdth driver or the kernel and it renders this neat controller rather ineffective.

Regards,
-mjy
[please CC: answers]

#1 lore raw
Quoted 7 lines · Marinos Yannikos

Hi,

our brand-new ICP GDT8523RZ controller with 6 disks peaks out at 45MB/s under 2.4.10, while with 2.2.19 it reaches 85MB/s (seq. read performance). It should realistically be able to reach at least 150-200MB/s in this configuration

Well that's pretty bad, isn't it?

Some things which it would be interesting to try out:

- Disable CONFIG_HIGHMEM in your kernel config, see what that does.

- Try linux-2.4.13 -- 2.4.10 was a bit of a dog (this may make a difference if the driver uese the new PCI DMA API, but it doesn't explain why 2.2.x does so much better).

- Profile the kernel. You may enable profiling by booting the kernel with the LILO option `profile=1'.

Then, making sure that /boot/System.map reflects the running kernel, run this little script

#!/bin/sh
TIMEFILE=/tmp/$(basename $1).time
sudo readprofile -r
time "$@"
readprofile -v -m /boot/System.map | sort -n +2 | tail -40 | tee $TIMEFILE echo created $TIMEFILE

With something like:

~/kern-prof.sh cp some_huge_file /dev/null

#2 lore raw
In reply to Marinos Yannikos
Quoted 1 line · Andrew Morton

Well that's pretty bad, isn't it?

We could have bought a much cheaper controller and slower disks ;-)

Quoted 1 line

- Disable CONFIG_HIGHMEM

That seems to have no effect on performance. Btw., the 64GB support in 2.4.10 seemed to be buggy ("0-order allocation failed", then the DB crashed), so we were using the 4GB setting.

Quoted 1 line

- Try linux-2.4.13

This helped - now performance is up to par with 2.2.19 (~ 85MB/s) - thanks!

Quoted 2 lines

- Profile the kernel. [...] With something like:
~/kern-prof.sh cp some_huge_file /dev/null

I tried this, but with
dd if=/dev/sda of=/dev/null bs=1024k count=3000

(the fs is too slow - so "cp" peaks out at 17MB/s!)

The result (last 4 lines):
c01388fc try_to_free_buffers 55 0.1511

c0128b10 file_read_actor                            1179  14.0357
c01053b0 default_idle                               6784 130.4615
00000000 total                                      8695   0.0065

Does this suggest that the kernel isn't the bottleneck?

Regards,
Marinos

Signature · 4 lines
Marinos Yannikos, CEO
Preisvergleich Internet Services AG
Franzensbrückenstraße 8/2/16, A-1020 Wien
Tel./Fax: (+431) 5811609-52/-55
#3 lore raw
In reply to Marinos Yannikos
Quoted 6 lines · Marinos J. Yannikos
Quoted 1 line · Andrew Morton

Well that's pretty bad, isn't it?

We could have bought a much cheaper controller and slower disks ;-)

It's Linux's way of saving you money :)

Quoted 5 lines
Quoted 1 line

- Disable CONFIG_HIGHMEM

That seems to have no effect on performance. Btw., the 64GB support in 2.4.10 seemed to be buggy ("0-order allocation failed", then the DB crashed), so we were using the 4GB setting.

There have been many reports of this happening, but they are tapering off now.

Quoted 3 lines
Quoted 1 line

- Try linux-2.4.13

This helped - now performance is up to par with 2.2.19 (~ 85MB/s) - thanks!

It's surprising that 2.4.10->2.4.13 actually doubled throughput.

Quoted 7 lines
Quoted 2 lines

- Profile the kernel. [...] With something like:
~/kern-prof.sh cp some_huge_file /dev/null

I tried this, but with
dd if=/dev/sda of=/dev/null bs=1024k count=3000

(the fs is too slow - so "cp" peaks out at 17MB/s!)

Really? Are you saying that on a controller which can do 85 megs/second, you can't read files through the filesystem at greater than 17? Which filesystem?

Quoted 5 lines

The result (last 4 lines):
c01388fc try_to_free_buffers 55 0.1511

c0128b10 file_read_actor                            1179  14.0357
c01053b0 default_idle                               6784 130.4615
00000000 total                                      8695   0.0065

OK, that's normal and proper. Almost all the kernel time is spent copying data.

Quoted 1 line

Does this suggest that the kernel isn't the bottleneck?

Well... We seem to have three issues here:

1: Why isn't the controller achieving the manufacturer's
claimed throughput?

Don't know. Maybe it's the software copy. Maybe it's the device driver. Maybe they lied :). It'd be interesting to test it on the same machine with the vendor's drivers and win2k.

2: 0-order allocation failures.

3: Poor `cp' throughput. This one is strange. Perhaps
`cp' is using a small transfer-size and the kernel's readhead isn't working properly. Could you experiment with this some more? For example, what happens with

	dd if=large_file of=/dev/null bs=4096k

-

#4 lore raw
In reply to Marinos Yannikos
Quoted 2 lines · Andrew Morton

[...] you can't read files through the filesystem
at greater than 17? Which filesystem?

ext2 (on which I did a "tune2fs -j", but then I found out that the kernel apparently doesn't support ext3). But, see below.

Quoted 3 lines

[...] It'd be interesting to test it on the same machine with

the vendor's drivers and win2k.

Indeed - can't do so, unfortunately. It could also be an issue with the RAID-5 configuration (block size and arrangement of the disks in the array could be less than optimal as suggested by the ICP BIOS), so I'm in touch with the very helpful ICP support people as well.

Quoted 1 line
	dd if=large_file of=/dev/null bs=4096k

OK, that's quite odd - now I get a reasonable ~55 MB/s whatever I try (dd, cp), so that must have been my mistake (or due to the kernel profiling option?).

By the way, I'm having lock-ups with 2.4.13 that I can reproduce (vi <file on reiserfs partition> ... :q ... "Segmentation fault" ... lock-up - console input still works, but apparently processes don't get scheduled).

Regards,
Marinos

Signature · 4 lines
Marinos Yannikos, CEO
Preisvergleich Internet Services AG
Franzensbrückenstraße 8/2/16, A-1020 Wien
Tel./Fax: (+431) 5811609-52/-55