Strange IO stats on RAID1?

DISCUSSION Paul Waldo 2 participants 3 messages Aug 30, 2006 – Sep 4, 2006 MD RAID

Share & tools

Paul Waldo

author

2006-08-30 12:09 UTC #0 lore raw

Hi all,

I have an IMAP mail server where mail messages are stored on a RAID1 array. The access on that array (/dev/md3) has seemed slow, so I did some investigating. "iostat -x /dev/hd[bd] /dev/md3" shows this:

avg-cpu: %user %nice %system %iowait %steal %idle

         14.34   47.03   13.99   19.64    0.00    5.00

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
hdb          1.05  34.88 35.48  2.60 1531.23  299.85   765.62   149.93    48.08     0.20    5.38   3.07  11.69
hdd          0.70  34.83 41.08  2.65  963.12  299.85   481.56   149.93    28.88     0.15    3.49   1.65   7.24
md3          0.00   0.00 78.31 36.98 2494.35  295.85  1247.18   147.93    24.20     0.00    0.00   0.00   0.00

Here is my /proc/mdstat for md3:

md3 : active raid1 hdb1[0] hdd1[1]

     12699264 blocks [2/2] [UU]

Observations:
- Stats for raw disks writes are similar
- Stats for reads on hdd are about half that of hdd

I may be (or probably am) way off base on this, but I would think that read requests to each disk would be balanced, providing roughly the same level of read activity on each disk. Is that correct? Are there any tuning suggestions for increasing performance?

Thanks in advance!

Paul

Neil Brown

2006-09-04 05:28 UTC #1 lore raw

Quoted 19 lines · On Wednesday August 30, [email protected] wrote:

Hi all,
I have an IMAP mail server where mail messages are stored on a RAID1 array. The access on that array (/dev/md3) has seemed slow, so I did some investigating. "iostat -x /dev/hd[bd] /dev/md3" shows this:
avg-cpu: %user %nice %system %iowait %steal %idle
          14.34   47.03   13.99   19.64    0.00    5.00
Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
hdb          1.05  34.88 35.48  2.60 1531.23  299.85   765.62   149.93    48.08     0.20    5.38   3.07  11.69
hdd          0.70  34.83 41.08  2.65  963.12  299.85   481.56   149.93    28.88     0.15    3.49   1.65   7.24
md3          0.00   0.00 78.31 36.98 2494.35  295.85  1247.18   147.93    24.20     0.00    0.00   0.00   0.00
Here is my /proc/mdstat for md3:

Quoted 7 lines

md3 : active raid1 hdb1[0] hdd1[1]
      12699264 blocks [2/2] [UU]
Observations:
- Stats for raw disks writes are similar
- Stats for reads on hdd are about half that of hdd

Yes.... odd.

There shouldn't be any bias to one drive or the other. When it scans for the best drive to use, it always starts from the last one that was used, so there should be no bias to one or the other.

I assume they are identical drives?
dd if=/dev/hdXX of=/dev/null bs=1024k count=1024

give similar speeds for each?

NeilBrown

Signature · 1 line

VGER BF report: U 0.499857

Paul Waldo

author

2006-09-04 09:58 UTC #2 lore raw

Quoted 38 lines · Neil Brown

Quoted 22 lines · On Wednesday August 30, [email protected] wrote:
Hi all,
I have an IMAP mail server where mail messages are stored on a RAID1 array. The access on that array (/dev/md3) has seemed slow, so I did some investigating. "iostat -x /dev/hd[bd] /dev/md3" shows this:
avg-cpu: %user %nice %system %iowait %steal %idle
          14.34   47.03   13.99   19.64    0.00    5.00
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s
avgrq-sz avgqu-sz await svctm %util hdb 1.05 34.88 35.48
2.60 1531.23 299.85 765.62 149.93 48.08 0.20 5.38 3.07
11.69 hdd 0.70 34.83 41.08 2.65 963.12 299.85 481.56
149.93 28.88 0.15 3.49 1.65 7.24 md3 0.00 0.00
78.31 36.98 2494.35 295.85 1247.18 147.93 24.20 0.00 0.00
0.00 0.00
Here is my /proc/mdstat for md3:
Quoted 7 lines
md3 : active raid1 hdb1[0] hdd1[1]
      12699264 blocks [2/2] [UU]
Observations:
- Stats for raw disks writes are similar
- Stats for reads on hdd are about half that of hdd
Yes.... odd.
There shouldn't be any bias to one drive or the other. When it scans for the best drive to use, it always starts from the last one that was used, so there should be no bias to one or the other.

Quoted 6 lines

I assume they are identical drives?
dd if=/dev/hdXX of=/dev/null bs=1024k count=1024
give similar speeds for each?
NeilBrown

Hi Neil,

They are different drives:

root@mail:~# cat /proc/ide/hd[bd]/model
WDC WD205BA
IC35L020AVER07-0

The read speeds are different, 14% by my calculations:

root@mail:~# dd if=/dev/hdb of=/dev/null bs=1024k count=1024
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 49.1775 seconds, 21.8 MB/s
root@mail:~# dd if=/dev/hdd of=/dev/null bs=1024k count=1024
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 42.3739 seconds, 25.3 MB/s

BTW, my kernel is 2.6.15-26 and mdadm is 1.12.0, running on Kubuntu 6.06. Thanks!

Paul

Signature · 1 line

VGER BF report: H 0.00374821