Skip to contentSkip to conversation

Slow I/O with SMP, Fusion-MPT and u160 SCSI JBOD

DISCUSSION Phil Dier 3 participants 5 messages Sep 14, 2005

lore
Share & tools
lore

Phil Dier

author
#0 lore raw

Hi,

I just tried the 2.6.14-rc1 kernel to see if it exhibits the behaviour I have described before[0]. It still does. Briefly, I have a dual Xeon 2.8 with 4GB ram, 1 adaptec and 1 fusion-mpt scsi device, each connected to 1 SCSI JBOD with 5 disks apiece[1]. I/O on scsi ids 0 and 1 both work flawlessly. I built an md device across ids 2 and 3, and the disks connected to the fusion-mpt are unbearably slow. This behaviour was not present in 2.6.12.3. When I partition the devices in question and mount a filesystem on them, they exhibit the same behaviour individually.

[0]: http://www.ussg.iu.edu/hypermail/linux/kernel/0508.3/1233.html
[1]: http://www.icglink.com/debug-2.6.14-rc1.html

Here is what iostat looks like during what should be heavy I/O (my tesh.sh script described in [1] is running), showing only the relevant devices:

$ iostat 1 -k | awk '/(Device|md5|md6|md7)/{print $0;}'
Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
md5               0.62         0.00         2.46        244     198312
md6               0.62         0.00         2.46        228     198312
md7               0.61         0.01         2.45        440     197856
Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
md5               0.00         0.00         0.00          0          0
md6               0.00         0.00         0.00          0          0
md7               0.00         0.00         0.00          0          0
Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
md5              25.74         0.00       102.97          0        104
md6              25.74         0.00       102.97          0        104
md7               9.90         0.00        39.60          0         40
Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
md5              59.41         0.00       237.62          0        240
md6              60.40         3.96       237.62          4        240
md7              20.79         3.96        79.21          4         80
Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
md5             100.99         0.00       403.96          0        408
md6             100.99         0.00       403.96          0        408
md7              72.28         0.00       289.11          0        292
Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
md5           16242.57         0.00     64970.30          0      65620
md6           16242.57         0.00     64970.30          0      65620
md7           16230.69         0.00     64922.77          0      65572
Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
md5               0.00         0.00         0.00          0          0
md6               0.00         0.00         0.00          0          0
md7               0.00         0.00         0.00          0          0
Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
md5            1014.85         0.00      4059.41          0       4100
md6            1014.85         0.00      4059.41          0       4100
md7            1014.85         0.00      4059.41          0       4100
Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
md5               0.00         0.00         0.00          0          0
md6               0.00         0.00         0.00          0          0
md7               0.00         0.00         0.00          0          0
Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
md5               0.00         0.00         0.00          0          0
md6               0.00         0.00         0.00          0          0
md7               0.00         0.00         0.00          0          0
Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
md5               0.00         0.00         0.00          0          0
md6               0.00         0.00         0.00          0          0
md7               0.00         0.00         0.00          0          0
Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
md5               0.00         0.00         0.00          0          0
md6               0.00         0.00         0.00          0          0
md7               0.00         0.00         0.00          0          0
Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
md5               0.00         0.00         0.00          0          0
md6               0.00         0.00         0.00          0          0
md7               0.00         0.00         0.00          0          0
Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
md5               0.00         0.00         0.00          0          0
md6               0.00         0.00         0.00          0          0
md7               0.00         0.00         0.00          0          0
Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
md5               0.00         0.00         0.00          0          0
md6               0.00         0.00         0.00          0          0
md7               0.00         0.00         0.00          0          0
Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
md5               0.00         0.00         0.00          0          0
md6               0.00         0.00         0.00          0          0
md7               0.00         0.00         0.00          0          0
Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
md5               0.00         0.00         0.00          0          0
md6               0.00         0.00         0.00          0          0
md7               0.00         0.00         0.00          0          0
Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
md5               0.00         0.00         0.00          0          0
md6               0.00         0.00         0.00          0          0
md7               0.00         0.00         0.00          0          0
Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
md5               0.00         0.00         0.00          0          0
md6               0.00         0.00         0.00          0          0
md7               0.00         0.00         0.00          0          0
Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
md5               0.00         0.00         0.00          0          0
md6               0.00         0.00         0.00          0          0
md7               0.00         0.00         0.00          0          0
Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
md5               0.00         0.00         0.00          0          0
md6               0.00         0.00         0.00          0          0
md7               0.00         0.00         0.00          0          0
Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
md5               0.00         0.00         0.00          0          0
md6               0.00         0.00         0.00          0          0
md7               0.00         0.00         0.00          0          0
Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
md5               0.00         0.00         0.00          0          0
md6               0.00         0.00         0.00          0          0
md7               0.00         0.00         0.00          0          0
Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
md5               0.00         0.00         0.00          0          0
md6               0.00         0.00         0.00          0          0
md7               0.00         0.00         0.00          0          0
Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
md5               0.00         0.00         0.00          0          0
md6               0.00         0.00         0.00          0          0
md7               0.00         0.00         0.00          0          0
Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
md5               0.00         0.00         0.00          0          0
md6               0.00         0.00         0.00          0          0
md7               0.00         0.00         0.00          0          0
Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
md5               0.00         0.00         0.00          0          0
md6               0.00         0.00         0.00          0          0
md7               0.00         0.00         0.00          0          0
Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
md5               0.00         0.00         0.00          0          0
md6               0.00         0.00         0.00          0          0
md7               0.00         0.00         0.00          0          0
Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
md5               0.00         0.00         0.00          0          0
md6               0.00         0.00         0.00          0          0
md7               0.00         0.00         0.00          0          0
Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
md5               0.00         0.00         0.00          0          0
md6               0.00         0.00         0.00          0          0
md7               0.00         0.00         0.00          0          0
Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
md5               0.00         0.00         0.00          0          0
md6               0.00         0.00         0.00          0          0
md7               0.00         0.00         0.00          0          0
Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
md5               0.00         0.00         0.00          0          0
md6               0.00         0.00         0.00          0          0
md7               0.00         0.00         0.00          0          0
Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
md5               0.00         0.00         0.00          0          0
md6               0.00         0.00         0.00          0          0
md7               0.00         0.00         0.00          0          0
Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
md5               0.00         0.00         0.00          0          0
md6               0.00         0.00         0.00          0          0
md7               0.00         0.00         0.00          0          0
Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
md5               0.00         0.00         0.00          0          0
md6               0.00         0.00         0.00          0          0
md7               0.00         0.00         0.00          0          0
Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
md5            3298.02         0.00     13192.08          0      13324
md6            3298.02         0.00     13192.08          0      13324
md7            3298.02         0.00     13192.08          0      13324
Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
md5               0.00         0.00         0.00          0          0
md6               0.00         0.00         0.00          0          0
md7               0.00         0.00         0.00          0          0
Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
md5               0.00         0.00         0.00          0          0
md6               0.00         0.00         0.00          0          0
md7               0.00         0.00         0.00          0          0
Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
md5               0.00         0.00         0.00          0          0
md6               0.00         0.00         0.00          0          0
md7               0.00         0.00         0.00          0          0
Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
md5            2789.11         0.00     11156.44          0      11268
md6            2789.11         0.00     11156.44          0      11268
md7            2789.11         0.00     11156.44          0      11268
Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
md5               0.00         0.00         0.00          0          0
md6               0.00         0.00         0.00          0          0
md7               0.00         0.00         0.00          0          0
Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
md5               0.00         0.00         0.00          0          0
md6               0.00         0.00         0.00          0          0
md7               0.00         0.00         0.00          0          0
Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
md5               0.00         0.00         0.00          0          0
md6               0.00         0.00         0.00          0          0
md7               0.00         0.00         0.00          0          0

Thanks for looking. Please cc me on replies.

Signature · 3 lines
Phil Dier (ICGLink.com -- 615 370-1530 x733)

/* vim:set noai nocindent ts=8 sw=8: */
#1 lore raw

Phil,

What happens if you pull 1gb or 2gb of memory from your box? Running a news feed shouldn't be memory intensive and maybe you've run into a low-mem vs. high-mem issue. It would be an interesting test in any case.

John

#2 lore raw
In reply to Phil Dier
Quoted 4 lines · Phil Dier

Hi,

I just tried the 2.6.14-rc1 kernel to see if it exhibits the behaviour I have described before[0]. It still does. Briefly, I have a dual Xeon

..

Do you still have HZ set to 1000 in your .config file? (as per 2.6.12)

author
#3 lore raw
Quoted 8 lines · On Wed, 14 Sep 2005 17:41:58 -0400 Mark Lord <[email protected]> wrote:
Quoted 4 lines · Phil Dier

Hi,

I just tried the 2.6.14-rc1 kernel to see if it exhibits the behaviour I have described before[0]. It still does. Briefly, I have a dual Xeon

..

Do you still have HZ set to 1000 in your .config file? (as per 2.6.12)

No, the config for .14 is using the default of 100. I'm recompiling with HZ=1000 to see if that makes a difference.

Signature · 3 lines
Phil Dier (ICGLink.com -- 615 370-1530 x733)

/* vim:set noai nocindent ts=8 sw=8: */
author
#4 lore raw
In reply to Mark Lord
Quoted 8 lines · On Wed, 14 Sep 2005 17:41:58 -0400 Mark Lord <[email protected]> wrote:
Quoted 4 lines · Phil Dier

Hi,

I just tried the 2.6.14-rc1 kernel to see if it exhibits the behaviour I have described before[0]. It still does. Briefly, I have a dual Xeon

..

Do you still have HZ set to 1000 in your .config file? (as per 2.6.12)

With HZ at 1000, the speeds seem to be fine. Why would this occur with just this one controller? Just driver differences?

Signature · 3 lines
Phil Dier (ICGLink.com -- 615 370-1530 x733)

/* vim:set noai nocindent ts=8 sw=8: */