bad performance on RAID 5

DISCUSSION Sevrin Robstad 9 participants 16 messages Jan 17, 2007 – Jan 21, 2007 MD RAID

lore

Share & tools

lore

Sevrin Robstad

author

2007-01-17 21:33 UTC #0 lore raw

I'm suffering from bad performance on my RAID5.

a "echo check >/sys/block/md0/md/sync_action"

gives a speed at only about 5000K/sec , and HIGH load average :

# uptime
20:03:55 up 8 days, 19:55,  1 user,  load average: 11.70, 4.04, 1.52

kernel is 2.6.18.1.2257.fc5
mdadm is v2.5.5

the system consist of an athlon XP1,2GHz and two Sil3114 4port S-ATA PCI cards with a total of 6 250gb S-ATA drives connected.

[root@compaq ~]# mdadm --detail /dev/md0
/dev/md0:

       Version : 00.90.03

Creation Time : Tue Dec 5 00:33:01 2006
Raid Level : raid5
Array Size : 1218931200 (1162.46 GiB 1248.19 GB)
Device Size : 243786240 (232.49 GiB 249.64 GB)
Raid Devices : 6
Total Devices : 6
Preferred Minor : 0
Persistence : Superblock is persistent

Update Time : Wed Jan 17 23:14:39 2007
State : clean
Active Devices : 6
Working Devices : 6
Failed Devices : 0
Spare Devices : 0

Layout : left-symmetric
Chunk Size : 256K

UUID : 27dce477:6f45d11b:77377d08:732fa0e6
Events : 0.58

Number Major Minor RaidDevice State

      0       8        1        0      active sync   /dev/sda1
      1       8       17        1      active sync   /dev/sdb1
      2       8       33        2      active sync   /dev/sdc1
      3       8       49        3      active sync   /dev/sdd1
      4       8       65        4      active sync   /dev/sde1
      5       8       81        5      active sync   /dev/sdf1

[root@compaq ~]#

Sevrin

Justin Piszcz

2007-01-17 22:38 UTC #1 lore raw

Quoted 46 lines · Sevrin Robstad

I'm suffering from bad performance on my RAID5.
a "echo check >/sys/block/md0/md/sync_action"
gives a speed at only about 5000K/sec , and HIGH load average :
# uptime
20:03:55 up 8 days, 19:55,  1 user,  load average: 11.70, 4.04, 1.52
kernel is 2.6.18.1.2257.fc5
mdadm is v2.5.5
the system consist of an athlon XP1,2GHz and two Sil3114 4port S-ATA PCI cards with a total of 6 250gb S-ATA drives connected.
[root@compaq ~]# mdadm --detail /dev/md0
/dev/md0:
       Version : 00.90.03
Creation Time : Tue Dec 5 00:33:01 2006
Raid Level : raid5
Array Size : 1218931200 (1162.46 GiB 1248.19 GB)
Device Size : 243786240 (232.49 GiB 249.64 GB)
Raid Devices : 6
Total Devices : 6
Preferred Minor : 0
Persistence : Superblock is persistent
Update Time : Wed Jan 17 23:14:39 2007
State : clean
Active Devices : 6
Working Devices : 6
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 256K
UUID : 27dce477:6f45d11b:77377d08:732fa0e6
Events : 0.58
Number Major Minor RaidDevice State
      0       8        1        0      active sync   /dev/sda1
      1       8       17        1      active sync   /dev/sdb1
      2       8       33        2      active sync   /dev/sdc1
      3       8       49        3      active sync   /dev/sdd1
      4       8       65        4      active sync   /dev/sde1
      5       8       81        5      active sync   /dev/sdf1

Quoted 8 lines

[root@compaq ~]#
Sevrin
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

If they are on the PCI bus, that is about right, you probably should be getting 10-15MB/s, but it is about right. If you had each drive on its own PCI-e controller, then you would get much faster speeds.

Raz Ben-Jehuda(caro)

2007-01-18 09:06 UTC #2 lore raw

did u increase the stripe cache size ?

Quoted 66 lines · Justin Piszcz

Quoted 47 lines · Sevrin Robstad
I'm suffering from bad performance on my RAID5.
a "echo check >/sys/block/md0/md/sync_action"
gives a speed at only about 5000K/sec , and HIGH load average :
# uptime
20:03:55 up 8 days, 19:55,  1 user,  load average: 11.70, 4.04, 1.52
kernel is 2.6.18.1.2257.fc5
mdadm is v2.5.5
the system consist of an athlon XP1,2GHz and two Sil3114 4port S-ATA PCI cards with a total of 6 250gb S-ATA drives connected.
[root@compaq ~]# mdadm --detail /dev/md0
/dev/md0:
       Version : 00.90.03
Creation Time : Tue Dec 5 00:33:01 2006
Raid Level : raid5
Array Size : 1218931200 (1162.46 GiB 1248.19 GB)
Device Size : 243786240 (232.49 GiB 249.64 GB)
Raid Devices : 6
Total Devices : 6
Preferred Minor : 0
Persistence : Superblock is persistent
Update Time : Wed Jan 17 23:14:39 2007
State : clean
Active Devices : 6
Working Devices : 6
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 256K
UUID : 27dce477:6f45d11b:77377d08:732fa0e6
Events : 0.58
Number Major Minor RaidDevice State
      0       8        1        0      active sync   /dev/sda1
      1       8       17        1      active sync   /dev/sdb1
      2       8       33        2      active sync   /dev/sdc1
      3       8       49        3      active sync   /dev/sdd1
      4       8       65        4      active sync   /dev/sde1
      5       8       81        5      active sync   /dev/sdf1
Quoted 8 lines
[root@compaq ~]#
Sevrin
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
If they are on the PCI bus, that is about right, you probably should be getting 10-15MB/s, but it is about right. If you had each drive on its own PCI-e controller, then you would get much faster speeds.
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Signature · 1 line

Raz

Sevrin Robstad

author

2007-01-18 18:20 UTC #3 lore raw

I've tried to increase the cache size - I can't measure any difference.....

Quoted 73 lines · Raz Ben-Jehuda(caro)

did u increase the stripe cache size ?
Quoted 67 lines · Justin Piszcz
Quoted 47 lines · Sevrin Robstad
I'm suffering from bad performance on my RAID5.
a "echo check >/sys/block/md0/md/sync_action"
gives a speed at only about 5000K/sec , and HIGH load average :
# uptime
20:03:55 up 8 days, 19:55,  1 user,  load average: 11.70, 4.04, 1.52
kernel is 2.6.18.1.2257.fc5
mdadm is v2.5.5
the system consist of an athlon XP1,2GHz and two Sil3114 4port S-ATA PCI cards with a total of 6 250gb S-ATA drives connected.
[root@compaq ~]# mdadm --detail /dev/md0
/dev/md0:
       Version : 00.90.03
Creation Time : Tue Dec 5 00:33:01 2006
Raid Level : raid5
Array Size : 1218931200 (1162.46 GiB 1248.19 GB)
Device Size : 243786240 (232.49 GiB 249.64 GB)
Raid Devices : 6
Total Devices : 6
Preferred Minor : 0
Persistence : Superblock is persistent
Update Time : Wed Jan 17 23:14:39 2007
State : clean
Active Devices : 6
Working Devices : 6
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 256K
UUID : 27dce477:6f45d11b:77377d08:732fa0e6
Events : 0.58
Number Major Minor RaidDevice State
      0       8        1        0      active sync   /dev/sda1
      1       8       17        1      active sync   /dev/sdb1
      2       8       33        2      active sync   /dev/sdc1
      3       8       49        3      active sync   /dev/sdd1
      4       8       65        4      active sync   /dev/sde1
      5       8       81        5      active sync   /dev/sdf1
Quoted 6 lines
[root@compaq ~]#
Sevrin
-
To unsubscribe from this list: send the line "unsubscribe
linux-raid" in
Quoted 2 lines
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
If they are on the PCI bus, that is about right, you probably should be getting 10-15MB/s, but it is about right. If you had each drive on its own PCI-e controller, then you would get much faster speeds.
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Raz Ben-Jehuda(caro)

2007-01-18 19:56 UTC #4 lore raw

iorder to understand what is going in your system you should:
1. determine the access pattern to the volume. meaning:
sequetial ? random access ?
sync io ? async io ?
mostly read ? mostly write ?
Are you using small buffers ? big buffers ?

2. you should test the controller capabilty.
meaning :
see if dd'in for each disk in the system seperately reduces the total
throughput.

Quoted 78 lines · Sevrin Robstad

I've tried to increase the cache size - I can't measure any difference.....
Quoted 73 lines · Raz Ben-Jehuda(caro)
did u increase the stripe cache size ?
Quoted 67 lines · Justin Piszcz
Quoted 47 lines · Sevrin Robstad
I'm suffering from bad performance on my RAID5.
a "echo check >/sys/block/md0/md/sync_action"
gives a speed at only about 5000K/sec , and HIGH load average :
# uptime
20:03:55 up 8 days, 19:55,  1 user,  load average: 11.70, 4.04, 1.52
kernel is 2.6.18.1.2257.fc5
mdadm is v2.5.5
the system consist of an athlon XP1,2GHz and two Sil3114 4port S-ATA PCI cards with a total of 6 250gb S-ATA drives connected.
[root@compaq ~]# mdadm --detail /dev/md0
/dev/md0:
       Version : 00.90.03
Creation Time : Tue Dec 5 00:33:01 2006
Raid Level : raid5
Array Size : 1218931200 (1162.46 GiB 1248.19 GB)
Device Size : 243786240 (232.49 GiB 249.64 GB)
Raid Devices : 6
Total Devices : 6
Preferred Minor : 0
Persistence : Superblock is persistent
Update Time : Wed Jan 17 23:14:39 2007
State : clean
Active Devices : 6
Working Devices : 6
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 256K
UUID : 27dce477:6f45d11b:77377d08:732fa0e6
Events : 0.58
Number Major Minor RaidDevice State
      0       8        1        0      active sync   /dev/sda1
      1       8       17        1      active sync   /dev/sdb1
      2       8       33        2      active sync   /dev/sdc1
      3       8       49        3      active sync   /dev/sdd1
      4       8       65        4      active sync   /dev/sde1
      5       8       81        5      active sync   /dev/sdf1
Quoted 6 lines
[root@compaq ~]#
Sevrin
-
To unsubscribe from this list: send the line "unsubscribe
linux-raid" in
Quoted 2 lines
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
If they are on the PCI bus, that is about right, you probably should be getting 10-15MB/s, but it is about right. If you had each drive on its own PCI-e controller, then you would get much faster speeds.
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Signature · 1 line

Raz

Bill Davidsen

2007-01-18 23:01 UTC #5 lore raw

In reply to Sevrin Robstad

Quoted 1 line · Sevrin Robstad

I've tried to increase the cache size - I can't measure any difference.....

You probably won't help small writes, but large writes will go faster with a stripe cache of size num_disks*chunk_size*2 or larger.

Quoted 67 lines

Quoted 65 lines · Raz Ben-Jehuda(caro)
did u increase the stripe cache size ?
Quoted 61 lines · Justin Piszcz
Quoted 47 lines · Sevrin Robstad
I'm suffering from bad performance on my RAID5.
a "echo check >/sys/block/md0/md/sync_action"
gives a speed at only about 5000K/sec , and HIGH load average :
# uptime
20:03:55 up 8 days, 19:55,  1 user,  load average: 11.70, 4.04, 1.52
kernel is 2.6.18.1.2257.fc5
mdadm is v2.5.5
the system consist of an athlon XP1,2GHz and two Sil3114 4port S-ATA PCI cards with a total of 6 250gb S-ATA drives connected.
[root@compaq ~]# mdadm --detail /dev/md0
/dev/md0:
       Version : 00.90.03
Creation Time : Tue Dec 5 00:33:01 2006
Raid Level : raid5
Array Size : 1218931200 (1162.46 GiB 1248.19 GB)
Device Size : 243786240 (232.49 GiB 249.64 GB)
Raid Devices : 6
Total Devices : 6
Preferred Minor : 0
Persistence : Superblock is persistent
Update Time : Wed Jan 17 23:14:39 2007
State : clean
Active Devices : 6
Working Devices : 6
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 256K
UUID : 27dce477:6f45d11b:77377d08:732fa0e6
Events : 0.58
Number Major Minor RaidDevice State
      0       8        1        0      active sync   /dev/sda1
      1       8       17        1      active sync   /dev/sdb1
      2       8       33        2      active sync   /dev/sdc1
      3       8       49        3      active sync   /dev/sdd1
      4       8       65        4      active sync   /dev/sde1
      5       8       81        5      active sync   /dev/sdf1
Quoted 6 lines
[root@compaq ~]#
Sevrin
-
To unsubscribe from this list: send the line "unsubscribe
linux-raid" in
Quoted 2 lines
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
If they are on the PCI bus, that is about right, you probably should be getting 10-15MB/s, but it is about right. If you had each drive on its own PCI-e controller, then you would get much faster speeds.

Signature · 3 lines

bill davidsen <[email protected]>
 CTO TMR Associates, Inc
 Doing interesting things with small computers since 1979

Sevrin Robstad

author

2007-01-18 18:19 UTC #6 lore raw

In reply to Justin Piszcz

Quoted 49 lines · Justin Piszcz

Quoted 49 lines
I'm suffering from bad performance on my RAID5.
a "echo check >/sys/block/md0/md/sync_action"
gives a speed at only about 5000K/sec , and HIGH load average :
# uptime
20:03:55 up 8 days, 19:55,  1 user,  load average: 11.70, 4.04, 1.52
kernel is 2.6.18.1.2257.fc5
mdadm is v2.5.5
the system consist of an athlon XP1,2GHz and two Sil3114 4port S-ATA PCI cards with a total of 6 250gb S-ATA drives connected.
[root@compaq ~]# mdadm --detail /dev/md0
/dev/md0:
       Version : 00.90.03
Creation Time : Tue Dec 5 00:33:01 2006
Raid Level : raid5
Array Size : 1218931200 (1162.46 GiB 1248.19 GB)
Device Size : 243786240 (232.49 GiB 249.64 GB)
Raid Devices : 6
Total Devices : 6
Preferred Minor : 0
Persistence : Superblock is persistent
Update Time : Wed Jan 17 23:14:39 2007
State : clean
Active Devices : 6
Working Devices : 6
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 256K
UUID : 27dce477:6f45d11b:77377d08:732fa0e6
Events : 0.58
Number Major Minor RaidDevice State
      0       8        1        0      active sync   /dev/sda1
      1       8       17        1      active sync   /dev/sdb1
      2       8       33        2      active sync   /dev/sdc1
      3       8       49        3      active sync   /dev/sdd1
      4       8       65        4      active sync   /dev/sde1
      5       8       81        5      active sync   /dev/sdf1
[root@compaq ~]#

Quoted 9 lines

Quoted 5 lines
Sevrin
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
If they are on the PCI bus, that is about right, you probably should be getting 10-15MB/s, but it is about right. If you had each drive on its own PCI-e controller, then you would get much faster speeds.

There are two controllers, 4 disks connected to one controller on the PCI-bus and 2 disks connected to the other controller. As you say I should have had 10-15MB/s, but I'm having 5MB *and* with really high load average.

Sevrin

Mark Hahn

2007-01-18 19:02 UTC #7 lore raw

Quoted 3 lines

Quoted 3 lines
Quoted 3 lines
gives a speed at only about 5000K/sec , and HIGH load average :
# uptime
20:03:55 up 8 days, 19:55,  1 user,  load average: 11.70, 4.04, 1.52

loadav is a bit misleading - it doesn't mean you had >11 runnable jobs. you might just have more jobs waiting on IO, being starved by the IO done by resync.

Quoted 1 line

Quoted 1 line
Quoted 1 line
Chunk Size : 256K

well, that's pretty big. it means 6*256K is necessary to do a whole-stripe update; your stripe cache may be too small to be effective.

Quoted 1 line

Quoted 1 line
If they are on the PCI bus, that is about right, you probably should be getting 10-15MB/s, but it is about right. If you had each drive on its own PCI-e controller, then you would get much faster speeds.

10-15 seems bizarrely low - one can certainly achieve >100 MB/s
over the PCI bus, so where does the factor of 6-10 come in?
seems like a R6 resync would do 4 reads and 2 writes for every
4 chunks of throughput (so should achieve more like 50 MB/s if the main limit is the bus at 100.)

Quoted 1 line

There are two controllers, 4 disks connected to one controller on the PCI-bus and 2 disks connected to the other controller.

well, you should probably look at the PCI topology ("lspci -v -t"),
and perhaps even the PCI settings (as well as stripe cache size, perhaps nr_requests, etc)

Sevrin Robstad

author

2007-01-18 20:47 UTC #8 lore raw

Quoted 11 lines · Mark Hahn

Quoted 1 line
Quoted 1 line
Chunk Size : 256K
well, that's pretty big. it means 6*256K is necessary to do a whole-stripe update; your stripe cache may be too small to be effective.
Quoted 1 line
Quoted 1 line
If they are on the PCI bus, that is about right, you probably should be getting 10-15MB/s, but it is about right. If you had each drive on its own PCI-e controller, then you would get much faster speeds.
10-15 seems bizarrely low - one can certainly achieve >100 MB/s
over the PCI bus, so where does the factor of 6-10 come in?
seems like a R6 resync would do 4 reads and 2 writes for every
4 chunks of throughput (so should achieve more like 50 MB/s if the main limit is the bus at 100.)

Quoted 6 lines

Quoted 1 line
There are two controllers, 4 disks connected to one controller on the PCI-bus and 2 disks connected to the other controller.
well, you should probably look at the PCI topology ("lspci -v -t"),
and perhaps even the PCI settings (as well as stripe cache size, perhaps nr_requests, etc)

I've tried to increase the stripe_cache_size a lot, no noticable difference. I set it to 8096....

[root@compaq md]# lspci -v -t
-[0000:00]-+-00.0 VIA Technologies, Inc. VT8366/A/7 [Apollo KT266/A/333]

          +-01.0-[0000:01]----00.0  ATI Technologies Inc Rage 128 RF/SG AGP
          +-05.0  Silicon Image, Inc. SiI 3114 [SATALink/SATARaid] 
Serial ATA Controller
          +-07.0  Silicon Image, Inc. SiI 3114 [SATALink/SATARaid] 
Serial ATA Controller
          +-08.0  Realtek Semiconductor Co., Ltd. RTL-8169 Gigabit Ethernet
          +-11.0  VIA Technologies, Inc. VT8233 PCI to ISA Bridge
          \-11.1  VIA Technologies, Inc.

VT82C586A/B/VT82C686/A/B/VT823x/A/C PIPC Bus Master IDE [root@compaq md]#

Steve Cousins

2007-01-18 18:29 UTC #9 lore raw

In reply to Sevrin Robstad

Quoted 5 lines · Sevrin Robstad

I'm suffering from bad performance on my RAID5.
a "echo check >/sys/block/md0/md/sync_action"
gives a speed at only about 5000K/sec , and HIGH load average :

What do you get when you try something like:

time dd if=/dev/zero of=/mount-point/test.dat bs=1024k count=1024

where /mount-point is where /dev/md0 is mounted.

This will create a 1 GiB file and it will tell you how long it takes to create it. Also, I'd try running Bonnie++ on it to see what the different performance values are.

I don't know a lot about the md sync process but I remember having my sync action stuck at a low value at one point and it didn't have anything to do with the performance of the RAID array in general.

Steve

Quoted 48 lines

# uptime
20:03:55 up 8 days, 19:55,  1 user,  load average: 11.70, 4.04, 1.52
kernel is 2.6.18.1.2257.fc5
mdadm is v2.5.5
the system consist of an athlon XP1,2GHz and two Sil3114 4port S-ATA PCI cards with a total of 6 250gb S-ATA drives connected.
[root@compaq ~]# mdadm --detail /dev/md0
/dev/md0:
       Version : 00.90.03
Creation Time : Tue Dec 5 00:33:01 2006
Raid Level : raid5
Array Size : 1218931200 (1162.46 GiB 1248.19 GB)
Device Size : 243786240 (232.49 GiB 249.64 GB)
Raid Devices : 6
Total Devices : 6
Preferred Minor : 0
Persistence : Superblock is persistent
Update Time : Wed Jan 17 23:14:39 2007
State : clean
Active Devices : 6
Working Devices : 6
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 256K
UUID : 27dce477:6f45d11b:77377d08:732fa0e6
Events : 0.58
Number Major Minor RaidDevice State
      0       8        1        0      active sync   /dev/sda1
      1       8       17        1      active sync   /dev/sdb1
      2       8       33        2      active sync   /dev/sdc1
      3       8       49        3      active sync   /dev/sdd1
      4       8       65        4      active sync   /dev/sde1
      5       8       81        5      active sync   /dev/sdf1
[root@compaq ~]#
Sevrin
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Signature · 4 lines

______________________________________________________________________
 Steve Cousins, Ocean Modeling Group    Email: [email protected]
 Marine Sciences, 452 Aubert Hall       http://rocky.umeoce.maine.edu
 Univ. of Maine, Orono, ME 04469        Phone: (207) 581-4302

Sevrin Robstad

author

2007-01-18 20:43 UTC #10 lore raw

Quoted 8 lines · Steve Cousins

Quoted 5 lines · Sevrin Robstad
I'm suffering from bad performance on my RAID5.
a "echo check >/sys/block/md0/md/sync_action"
gives a speed at only about 5000K/sec , and HIGH load average :
What do you get when you try something like:

Quoted 8 lines

time dd if=/dev/zero of=/mount-point/test.dat bs=1024k count=1024
where /mount-point is where /dev/md0 is mounted.
This will create a 1 GiB file and it will tell you how long it takes to create it. Also, I'd try running Bonnie++ on it to see what the different performance values are.
I don't know a lot about the md sync process but I remember having my sync action stuck at a low value at one point and it didn't have anything to do with the performance of the RAID array in general.

I ran the a couple of times, and got either about 28MB/s or about 34MB/s. strange. When I run the same test on a single disk connected to another controller I get about 60MB/s.

[root@compaq ~]# time dd if=/dev/zero of=/mnt/gigaraid/1gb.testfile bs=1024k cou nt=1024
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 38.1873 seconds, 28.1 MB/s

real    0m38.321s
user    0m0.009s
sys     0m8.602s

[root@compaq ~]# time dd if=/dev/zero of=/mnt/gigaraid/1gb.testfile bs=1024k cou nt=1024
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 31.4377 seconds, 34.2 MB/s

real    0m31.949s
user    0m0.016s
sys     0m8.988s

[root@compaq ~]# time dd if=/dev/zero of=/mnt/gigaraid/1gb.testfile bs=1024k cou nt=1024
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 37.599 seconds, 28.6 MB/s

real    0m38.151s
user    0m0.011s
sys     0m9.291s

[root@compaq ~]# time dd if=/dev/zero of=/mnt/gigaraid/1gb.testfile bs=1024k count=1024 1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 31.2569 seconds, 34.4 MB/s

real    0m31.765s
user    0m0.007s
sys     0m8.913s

[root@compaq ~]# time dd if=/dev/zero of=/mnt/gigaraid/1gb.testfile bs=1024k count=1024 1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 37.4778 seconds, 28.7 MB/s

real    0m37.923s
user    0m0.009s
sys     0m9.231s

Bill Davidsen

2007-01-18 23:10 UTC #11 lore raw

In reply to Steve Cousins

Quoted 10 lines · ) Steve Cousins

Quoted 5 lines · Sevrin Robstad
I'm suffering from bad performance on my RAID5.
a "echo check >/sys/block/md0/md/sync_action"
gives a speed at only about 5000K/sec , and HIGH load average :
What do you get when you try something like:
time dd if=/dev/zero of=/mount-point/test.dat bs=1024k count=1024

That doesn't give valid (repeatable) results due to caching issues. Go back to the thread I started on RAID-5 write, and see my results. More important, the way I got rid of the cache effects (beside an unloaded systems) was:
sync; time bash -c "dd if=/dev/zero bs=1024k count=2048
of=/mnt/point/file; sync"
I empty the cache, then time the dd including the sync at the end. Results are far more repeatable.

I actually was able to create custom arrays on unused devices to play with array setting, once you use the array you can't tune as much.

Quoted 53 lines

where /mount-point is where /dev/md0 is mounted.
This will create a 1 GiB file and it will tell you how long it takes to create it. Also, I'd try running Bonnie++ on it to see what the different performance values are.
I don't know a lot about the md sync process but I remember having my sync action stuck at a low value at one point and it didn't have anything to do with the performance of the RAID array in general.
Steve
Quoted 44 lines
# uptime
20:03:55 up 8 days, 19:55,  1 user,  load average: 11.70, 4.04, 1.52
kernel is 2.6.18.1.2257.fc5
mdadm is v2.5.5
the system consist of an athlon XP1,2GHz and two Sil3114 4port S-ATA PCI cards with a total of 6 250gb S-ATA drives connected.
[root@compaq ~]# mdadm --detail /dev/md0
/dev/md0:
       Version : 00.90.03
Creation Time : Tue Dec 5 00:33:01 2006
Raid Level : raid5
Array Size : 1218931200 (1162.46 GiB 1248.19 GB)
Device Size : 243786240 (232.49 GiB 249.64 GB)
Raid Devices : 6
Total Devices : 6
Preferred Minor : 0
Persistence : Superblock is persistent
Update Time : Wed Jan 17 23:14:39 2007
State : clean
Active Devices : 6
Working Devices : 6
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 256K
UUID : 27dce477:6f45d11b:77377d08:732fa0e6
Events : 0.58
Number Major Minor RaidDevice State
      0       8        1        0      active sync   /dev/sda1
      1       8       17        1      active sync   /dev/sdb1
      2       8       33        2      active sync   /dev/sdc1
      3       8       49        3      active sync   /dev/sdd1
      4       8       65        4      active sync   /dev/sde1
      5       8       81        5      active sync   /dev/sdf1
[root@compaq ~]#
Sevrin

Signature · 3 lines

bill davidsen <[email protected]>
 CTO TMR Associates, Inc
 Doing interesting things with small computers since 1979

Nix

2007-01-21 17:08 UTC #12 lore raw

On 18 Jan 2007, Bill Davidsen spake thusly:

Quoted 9 lines

Quoted 1 line · ) Steve Cousins
time dd if=/dev/zero of=/mount-point/test.dat bs=1024k count=1024
That doesn't give valid (repeatable) results due to caching issues. Go back to the thread I started on RAID-5 write, and see my results. More important, the way I got rid of the cache effects (beside an unloaded systems) was:
sync; time bash -c "dd if=/dev/zero bs=1024k count=2048 of=/mnt/point/file; sync"
I empty the cache, then time the dd including the sync at the
end. Results are far more repeatable.

Recent versions of dd have `oflag=direct' as well, to open the output with O_DIRECT. (I'm not sure what the state of O_DIRECT on regular files is though.)

Signature · 3 lines

`The serial comma, however, is correct and proper, and abandoning it will
surely lead to chaos, anarchy, rioting in the streets, the Terrorists
taking over, and possibly the complete collapse of Human Civilization.'

Bill Davidsen

2007-01-21 19:09 UTC #13 lore raw

Quoted 15 lines · Nix

On 18 Jan 2007, Bill Davidsen spake thusly:
Quoted 4 lines
Quoted 2 lines · ) Steve Cousins
time dd if=/dev/zero of=/mount-point/test.dat bs=1024k count=1024
Quoted 8 lines
That doesn't give valid (repeatable) results due to caching issues. Go back to the thread I started on RAID-5 write, and see my results. More important, the way I got rid of the cache effects (beside an unloaded systems) was:
sync; time bash -c "dd if=/dev/zero bs=1024k count=2048 of=/mnt/point/file; sync"
I empty the cache, then time the dd including the sync at the
end. Results are far more repeatable.

Quoted 5 lines

Recent versions of dd have `oflag=direct' as well, to open the output with O_DIRECT. (I'm not sure what the state of O_DIRECT on regular files is though.)

Doing the write using page cache and then just a single sync at the end gives a closer estimate of what can be written to the array in general. By going to O_DIRECT every i/o take place unbuffered, which is more typical of database use or similar. My original problem was capturing real data at 75MB/s and not being able to write it to an array that fast, taking advantage of page cache. By going to huge stripe cache, about 20x larger than default, performance was boosted to a more useful level.

I think both measurements are useful, but they aren't measuring the same thing.

Signature · 3 lines

bill davidsen <[email protected]>
 CTO TMR Associates, Inc
 Doing interesting things with small computers since 1979

dean gaudet

2007-01-19 01:33 UTC #14 lore raw

In reply to Sevrin Robstad

Quoted 8 lines · Sevrin Robstad

I'm suffering from bad performance on my RAID5.
a "echo check >/sys/block/md0/md/sync_action"
gives a speed at only about 5000K/sec , and HIGH load average :
# uptime
20:03:55 up 8 days, 19:55,  1 user,  load average: 11.70, 4.04, 1.52

iostat -kx /dev/sd? 10 ... and sum up the total IO...

also try increasing sync_speed_min/max....

and a loadavg jump like that suggests to me you have other things competing for the disk at the same time as the "check".

-dean

Roger Lucas

2007-01-20 10:18 UTC #15 lore raw

Hi Sevrin,

Are you sure all the disks are working OK?

We saw a problem here with a 4-disk SATA array. The array was working without errors but was _very_ slow. When we checked each disk individually, we found one disk was reporting SMART errors and running slow. We removed it and replaced it with a new disk and the RAID array ran at full speed again. Further tests on the removed disk found that it was having a lot of problems but just about hanging in there (although taking a long time for each operation) - hence the RAID array didn't mark it as faulty.

I would check the SMART logs with smartctl to see if anything looks a bit wrong and try benchmarking the disks individually too.

- Roger