Skip to contentSkip to conversation

RAID 5 write performance advice

DISCUSSION Mirko Benz 4 participants 14 messages Aug 24, 2005 – Sep 1, 2005 MD RAID

lore
Share & tools
lore

Mirko Benz

author
#0 lore raw

Hello,

We have recently tested Linux 2.6.12 SW RAID versus HW Raid. For SW Raid we used Linux 2.6.12 with 8 Seagate SATA NCQ disks no spare on a Dual Xeon platform. For HW Raid we used a Arc-1120 SATA Raid controller and a Fibre Channel Raid System (Dual 2 Gb, Infortrend).

READ SW:877 ARC:693 IFT:366
(MB/s @64k BS using disktest with raw device)

Read SW Raid performance is better than HW Raid. The FC RAID is limited by the interface.

WRITE SW:140 ARC:371 IFT:352

For SW RAID 5 we needed to adjust the scheduling policy. By default we got only 60 MB/s. SW RAID 0 write performance @64k is 522 MB/s.

Based on the performance numbers it looks like Linux SW RAID reads every data element of a stripe + parity in parallel, performs xor operations and than writes the data back to disk in parallel.

The HW Raid controllers seem to be a bit smarter in this regard. When they encounter a large write with enough data for a full stripe they seem to spare the read and perform only the xor + write in parallel. Hence no seek is required and in can be closer to RAID0 write performance.

We have an application were large amounts of data need to be sequentially written to disk (e.g. 100 MB at once). The storage system has an USV so write caching can be utilized.

I would like to have an advice if write performance similar to HW Raid controllers is possible with Linux or if there is something else that we could apply.

Thanks in advance,
Mirko

#1 lore raw
Quoted 10 lines · Mirko Benz

Hello,

We have recently tested Linux 2.6.12 SW RAID versus HW Raid. For SW Raid we used Linux 2.6.12 with 8 Seagate SATA NCQ disks no spare on a Dual Xeon platform. For HW Raid we used a Arc-1120 SATA Raid controller and a Fibre Channel Raid System (Dual 2 Gb, Infortrend).

READ SW:877 ARC:693 IFT:366
(MB/s @64k BS using disktest with raw device)

Quoted 7 lines

Read SW Raid performance is better than HW Raid. The FC RAID is limited by the interface.

WRITE SW:140 ARC:371 IFT:352

For SW RAID 5 we needed to adjust the scheduling policy. By default we got only 60 MB/s. SW RAID 0 write performance @64k is 522 MB/s.

how u test and get these number?

what is u raid5 configuration? chunk size?

Quoted 9 lines

Based on the performance numbers it looks like Linux SW RAID reads every data element of a stripe + parity in parallel, performs xor operations and than writes the data back to disk in parallel.

The HW Raid controllers seem to be a bit smarter in this regard. When they encounter a large write with enough data for a full stripe they seem to spare the read and perform only the xor + write in parallel. Hence no seek is required and in can be closer to RAID0 write performance.

this is stripe write and linux MD have this.

Quoted 16 lines

We have an application were large amounts of data need to be sequentially written to disk (e.g. 100 MB at once). The storage system has an USV so write caching can be utilized.

I would like to have an advice if write performance similar to HW Raid controllers is possible with Linux or if there is something else that we could apply.

Thanks in advance,
Mirko

-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

author
#2 lore raw

Hello,

The RAID5 configuration is: 8 SATA disks, 8 port Marvel SATA PCI-X controller chip (SuperMicro board), Dual Xeon, 1 GB RAM, stripe size 64K, no spare disk.

Measurements are performed to the ram md device with:
disktest -PT -T30 -h1 -K8 -B65536 -ID /dev/md0
using the default stripe size (64K). 128K stripe size does not make a real difference.

We have also increased the RAID 5 stripe cache by setting NR_STRIPES to a larger value but without any perceptible effect.

If Linux uses "stripe write" why is it so much slower than HW Raid? Is it disabled by default?

8 disks: 7 data disks + parity @ 64k stripe size = 448k data per stripe
The request size was smaller (tested up to 256K) than the size of a stripe.
We have seen errors for larger request sizes (e.g. 1 MB). Does Linux require the request size to be larger than a stripe to take advantage of "stripe write"?

Regards,
Mirko

Quoted 51 lines · Ming Zhang
Quoted 4 lines · Mirko Benz

Hello,

We have recently tested Linux 2.6.12 SW RAID versus HW Raid. For SW Raid we used Linux 2.6.12 with 8 Seagate SATA NCQ disks no spare on a Dual Xeon platform. For HW Raid we used a Arc-1120 SATA Raid controller and a Fibre Channel Raid System (Dual 2 Gb, Infortrend).

Quoted 10 lines

READ SW:877 ARC:693 IFT:366
(MB/s @64k BS using disktest with raw device)

Read SW Raid performance is better than HW Raid. The FC RAID is limited by the interface.

WRITE SW:140 ARC:371 IFT:352

For SW RAID 5 we needed to adjust the scheduling policy. By default we got only 60 MB/s. SW RAID 0 write performance @64k is 522 MB/s.

how u test and get these number?

what is u raid5 configuration? chunk size?

Quoted 5 lines

Based on the performance numbers it looks like Linux SW RAID reads every data element of a stripe + parity in parallel, performs xor operations and than writes the data back to disk in parallel.

The HW Raid controllers seem to be a bit smarter in this regard. When they encounter a large write with enough data for a full stripe they seem to spare the read and perform only the xor + write in parallel. Hence no seek is required and in can be closer to RAID0 write performance.

this is stripe write and linux MD have this.

Quoted 13 lines

We have an application were large amounts of data need to be sequentially written to disk (e.g. 100 MB at once). The storage system has an USV so write caching can be utilized.

I would like to have an advice if write performance similar to HW Raid controllers is possible with Linux or if there is something else that we could apply.

Thanks in advance,
Mirko

-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#3 lore raw
Quoted 5 lines · Mirko Benz

Hello,

The RAID5 configuration is: 8 SATA disks, 8 port Marvel SATA PCI-X controller chip (SuperMicro board), Dual Xeon, 1 GB RAM, stripe size 64K, no spare disk.

u have good luck on this Marvel SATA. When I use it with supermicro, I got very bad performance even with RAID0.

Quoted 6 lines

Measurements are performed to the ram md device with:
disktest -PT -T30 -h1 -K8 -B65536 -ID /dev/md0
using the default stripe size (64K). 128K stripe size does not make a real difference.

when u run this command, also run "iostat 1" at another console and see how many read and write.

also run "time dd if=/dev/zero of=/dev/mdX bs=1M" and see speed u get and see what iostat tell u.

Quoted 85 lines

We have also increased the RAID 5 stripe cache by setting NR_STRIPES to a larger value but without any perceptible effect.

If Linux uses "stripe write" why is it so much slower than HW Raid? Is it disabled by default?

8 disks: 7 data disks + parity @ 64k stripe size = 448k data per stripe The request size was smaller (tested up to 256K) than the size of a stripe. We have seen errors for larger request sizes (e.g. 1 MB). Does Linux require the request size to be larger than a stripe to take advantage of "stripe write"?

Regards,
Mirko

Quoted 65 lines · Ming Zhang
Quoted 10 lines · Mirko Benz

Hello,

We have recently tested Linux 2.6.12 SW RAID versus HW Raid. For SW Raid we used Linux 2.6.12 with 8 Seagate SATA NCQ disks no spare on a Dual Xeon platform. For HW Raid we used a Arc-1120 SATA Raid controller and a Fibre Channel Raid System (Dual 2 Gb, Infortrend).

READ SW:877 ARC:693 IFT:366
(MB/s @64k BS using disktest with raw device)

Quoted 9 lines

Read SW Raid performance is better than HW Raid. The FC RAID is limited by the interface.

WRITE SW:140 ARC:371 IFT:352

For SW RAID 5 we needed to adjust the scheduling policy. By default we got only 60 MB/s. SW RAID 0 write performance @64k is 522 MB/s.

how u test and get these number?

what is u raid5 configuration? chunk size?

Quoted 10 lines

Based on the performance numbers it looks like Linux SW RAID reads every data element of a stripe + parity in parallel, performs xor operations and than writes the data back to disk in parallel.

The HW Raid controllers seem to be a bit smarter in this regard. When they encounter a large write with enough data for a full stripe they seem to spare the read and perform only the xor + write in parallel. Hence no seek is required and in can be closer to RAID0 write performance.

this is stripe write and linux MD have this.

Quoted 17 lines

We have an application were large amounts of data need to be sequentially written to disk (e.g. 100 MB at once). The storage system has an USV so write caching can be utilized.

I would like to have an advice if write performance similar to HW Raid controllers is possible with Linux or if there is something else that we could apply.

Thanks in advance,
Mirko

-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#4 lore raw
In reply to Mirko Benz
Quoted 10 lines · On Wednesday August 24, [email protected] wrote:

Hello,

The RAID5 configuration is: 8 SATA disks, 8 port Marvel SATA PCI-X controller chip (SuperMicro board), Dual Xeon, 1 GB RAM, stripe size 64K, no spare disk.

Measurements are performed to the ram md device with:
disktest -PT -T30 -h1 -K8 -B65536 -ID /dev/md0
using the default stripe size (64K). 128K stripe size does not make a real difference.

May I suggest you try creating a filesystem on the device and doing tests in the filesystem? I have found the raw device slower that filesystem access before, and a quick test shows writing to the filesystem (ext3) is about 4 times as fast as writing to /dev/md1 on a 6 drive raid5 array.

Quoted 3 lines

If Linux uses "stripe write" why is it so much slower than HW Raid? Is it disabled by default?

No, it is never disabled. However it can only work if raid5 gets a full stripe of data before being asked to flush that data. Writing to /dev/md0 directly may cause flushes too often.

Does your application actually require writing to the raw device, or will you be using a filesystem?

NeilBrown

author
#5 lore raw

Hello,

We intend to export a lvm/md volume via iSCSI or SRP using InfiniBand to remote clients. There is no local file system processing on the storage platform. The clients may have a variety of file systems including ext3, GFS.

Single disk write performance is: 58,5 MB/s. With large sequential write operations I would expect something like 90% of n-1 * single_disk_performance if stripe write can be utilized. So roughly 400 MB/s – which the HW RAID devices achieve.

RAID setup:
Personalities : [raid0] [raid5]
md0 : active raid5 sdi[7] sdh[6] sdg[5] sdf[4] sde[3] sdd[2] sdc[1] sdb[0] 1094035712 blocks level 5, 64k chunk, algorithm 2 [8/8] [UUUUUUUU]

We have assigned the deadline scheduler to every disk in the RAID. The default scheduler gives much lower results.

*** dd TEST ***

time dd if=/dev/zero of=/dev/md0 bs=1M
5329911808 bytes transferred in 28,086199 seconds (189769779 bytes/sec)

iostat 5 output:
avg-cpu: %user %nice %sys %iowait %idle
0,10 0,00 87,80 7,30 4,80

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn hda 0,00 0,00 0,00 0 0
sda 0,00 0,00 0,00 0 0
sdb 1976,10 1576,10 53150,60 7912 266816 sdc 2072,31 1478,88 53150,60 7424 266816 sdd 2034,06 1525,10 53150,60 7656 266816 sde 1988,05 1439,04 53147,41 7224 266800 sdf 1975,10 1499,60 53147,41 7528 266800 sdg 1383,07 1485,26 53145,82 7456 266792 sdh 1562,55 1311,55 53145,82 6584 266792 sdi 1586,85 1295,62 53145,82 6504 266792 sdj 0,00 0,00 0,00 0 0
sdk 0,00 0,00 0,00 0 0
sdl 0,00 0,00 0,00 0 0
sdm 0,00 0,00 0,00 0 0
sdn 0,00 0,00 0,00 0 0
md0 46515,54 0,00 372124,30 0 1868064

Comments: Large write should not see any read operations. But there are some???

*** disktest ***

disktest -w -PT -T30 -h1 -K8 -B512k -ID /dev/md0

| 2005/08/25-17:27:04 | STAT | 4072 | v1.1.12 | /dev/md0 | Write throughput: 160152507.7B/s (152.73MB/s), IOPS 305.7/s.
| 2005/08/25-17:27:05 | STAT | 4072 | v1.1.12 | /dev/md0 | Write throughput: 160694272.0B/s (153.25MB/s), IOPS 306.6/s.
| 2005/08/25-17:27:06 | STAT | 4072 | v1.1.12 | /dev/md0 | Write throughput: 160339606.6B/s (152.91MB/s), IOPS 305.8/s.

iostat 5 output:
avg-cpu: %user %nice %sys %iowait %idle
38,96 0,00 50,25 5,29 5,49

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn hda 0,00 0,00 0,00 0 0
sda 1,20 0,00 11,18 0 56
sdb 986,43 0,00 39702,99 0 198912
sdc 922,75 0,00 39728,54 0 199040
sdd 895,81 0,00 39728,54 0 199040
sde 880,84 0,00 39728,54 0 199040
sdf 839,92 0,00 39728,54 0 199040
sdg 842,91 0,00 39728,54 0 199040
sdh 1557,49 0,00 79431,54 0 397952
sdi 2246,71 0,00 104411,98 0 523104
sdj 0,00 0,00 0,00 0 0
sdk 0,00 0,00 0,00 0 0
sdl 0,00 0,00 0,00 0 0
sdm 0,00 0,00 0,00 0 0
sdn 0,00 0,00 0,00 0 0
md0 1550,70 0,00 317574,45 0 1591048

Comments:
Zero read requests – as it should be. But the write requests are not proportional. sdh and sdi have significantly more requests??? The write requests to the disks of the RAID should be 1/7 higher than to the md device.
But there are significantly more write operations.

All these operations are to the raw device. Setting up a ext3 fs we get around 127 MB/s with dd.

Any idea?

--Mirko

-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#6 lore raw
Quoted 11 lines · Mirko Benz

Hello,

We intend to export a lvm/md volume via iSCSI or SRP using InfiniBand to remote clients. There is no local file system processing on the storage platform. The clients may have a variety of file systems including ext3, GFS.

Single disk write performance is: 58,5 MB/s. With large sequential write operations I would expect something like 90% of n-1 * single_disk_performance if stripe write can be utilized. So roughly 400 MB/s – which the HW RAID devices achieve.

change to RAID0 and test to see if u controller will be a bottleneck.

Quoted 36 lines

RAID setup:
Personalities : [raid0] [raid5]
md0 : active raid5 sdi[7] sdh[6] sdg[5] sdf[4] sde[3] sdd[2] sdc[1] sdb[0] 1094035712 blocks level 5, 64k chunk, algorithm 2 [8/8] [UUUUUUUU]

We have assigned the deadline scheduler to every disk in the RAID. The default scheduler gives much lower results.

*** dd TEST ***

time dd if=/dev/zero of=/dev/md0 bs=1M
5329911808 bytes transferred in 28,086199 seconds (189769779 bytes/sec)

iostat 5 output:
avg-cpu: %user %nice %sys %iowait %idle
0,10 0,00 87,80 7,30 4,80

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn hda 0,00 0,00 0,00 0 0
sda 0,00 0,00 0,00 0 0
sdb 1976,10 1576,10 53150,60 7912 266816 sdc 2072,31 1478,88 53150,60 7424 266816 sdd 2034,06 1525,10 53150,60 7656 266816 sde 1988,05 1439,04 53147,41 7224 266800 sdf 1975,10 1499,60 53147,41 7528 266800 sdg 1383,07 1485,26 53145,82 7456 266792 sdh 1562,55 1311,55 53145,82 6584 266792 sdi 1586,85 1295,62 53145,82 6504 266792 sdj 0,00 0,00 0,00 0 0
sdk 0,00 0,00 0,00 0 0
sdl 0,00 0,00 0,00 0 0
sdm 0,00 0,00 0,00 0 0
sdn 0,00 0,00 0,00 0 0
md0 46515,54 0,00 372124,30 0 1868064

Quoted 2 lines

Comments: Large write should not see any read operations. But there are some???

i always saw those small number reads and i feel it is reasonable since u stripe is 7 * 64KB

Quoted 4 lines

*** disktest ***

Quoted 8 lines

disktest -w -PT -T30 -h1 -K8 -B512k -ID /dev/md0

| 2005/08/25-17:27:04 | STAT | 4072 | v1.1.12 | /dev/md0 | Write
throughput: 160152507.7B/s (152.73MB/s), IOPS 305.7/s. | 2005/08/25-17:27:05 | STAT | 4072 | v1.1.12 | /dev/md0 | Write
throughput: 160694272.0B/s (153.25MB/s), IOPS 306.6/s. | 2005/08/25-17:27:06 | STAT | 4072 | v1.1.12 | /dev/md0 | Write
throughput: 160339606.6B/s (152.91MB/s), IOPS 305.8/s.

so here 152/7 = 21, large than what u sdc sdd got.

Quoted 23 lines

iostat 5 output:
avg-cpu: %user %nice %sys %iowait %idle
38,96 0,00 50,25 5,29 5,49

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn hda 0,00 0,00 0,00 0 0
sda 1,20 0,00 11,18 0 56
sdb 986,43 0,00 39702,99 0 198912
sdc 922,75 0,00 39728,54 0 199040
sdd 895,81 0,00 39728,54 0 199040
sde 880,84 0,00 39728,54 0 199040
sdf 839,92 0,00 39728,54 0 199040
sdg 842,91 0,00 39728,54 0 199040
sdh 1557,49 0,00 79431,54 0 397952
sdi 2246,71 0,00 104411,98 0 523104
sdj 0,00 0,00 0,00 0 0
sdk 0,00 0,00 0,00 0 0
sdl 0,00 0,00 0,00 0 0
sdm 0,00 0,00 0,00 0 0
sdn 0,00 0,00 0,00 0 0
md0 1550,70 0,00 317574,45 0 1591048

Quoted 3 lines

Comments:
Zero read requests – as it should be. But the write requests are not proportional. sdh and sdi have significantly more requests???

yes, interesting.

Quoted 11 lines

The write requests to the disks of the RAID should be 1/7 higher than to the md device.
But there are significantly more write operations.

All these operations are to the raw device. Setting up a ext3 fs we get around 127 MB/s with dd.

Any idea?

--Mirko

-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

author
#7 lore raw

Hello,

We have created a RAID 0 for the same environment:
Personalities : [raid0] [raid5]
md0 : active raid0 sdi[7] sdh[6] sdg[5] sdf[4] sde[3] sdd[2] sdc[1] sdb[0]

     1250326528 blocks 64k chunks

*** dd TEST ***

time dd if=/dev/zero of=/dev/md0 bs=1M
14967373824 bytes transferred in 32,060497 seconds (466847843 bytes/sec)

iostat 5 output:

avg-cpu:  %user   %nice    %sys %iowait   %idle
          0,00    0,00   89,60    9,50    0,90

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn

hda               0,00         0,00         0,00          0          0
sda               0,00         0,00         0,00          0          0
sdb             455,31         0,00    116559,52          0     581632
sdc             455,51         0,00    116540,28          0     581536
sdd             450,10         0,00    116545,09          0     581560
sde             454,11         0,00    116559,52          0     581632
sdf             452,30         0,00    116559,52          0     581632
sdg             454,71         0,00    116553,11          0     581600
sdh             453,31         0,00    116533,87          0     581504
sdi             453,91         0,00    116556,31          0     581616
sdj               0,00         0,00         0,00          0          0
sdk               0,00         0,00         0,00          0          0
sdl               0,00         0,00         0,00          0          0
sdm               0,00         0,00         0,00          0          0
sdn               0,00         0,00         0,00          0          0

md0 116556,11 0,00 932448,90 0 4652920

Comments: 466 MB / 8 = 58,25 MB/s which is about the same as a dd to a single disk (58,5 MB/s). So the controller + I/O subsystem is not the bottleneck.

Regards,
Mirko

#8 lore raw

seems i need to change the mail thread to

"why my RAID0 write speed so slow!" :P

i also use a Marvell 8 port PCI-X card. 8 SATA DISK RAID0, each single disk can give me around 55MB/s, but the RAID0 can only give me 203MB/s. I tried different io scheduler, all lead to same write speed at my side.

02:01.0 SCSI storage controller: Marvell MV88SX5081 8-port SATA I PCI-X Controller (rev 03)
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop-
ParErr- Stepping- SERR- FastB2B-
Status: Cap+ 66Mhz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort-
<TAbort- <MAbort- >SERR- <PERR-
Latency: 32, Cache Line Size 08
Interrupt: pin A routed to IRQ 24
Region 0: Memory at fa000000 (64-bit, non-prefetchable)
Capabilities: [40] Power Management version 2
Flags: PMEClk+ DSI- D1- D2- AuxCurrent=0mA PME
(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] Message Signalled Interrupts: 64bit+
Queue=0/0 Enable-
Address: 0000000000000000 Data: 0000
Capabilities: [60] PCI-X non-bridge device.
Command: DPERE- ERO- RBC=0 OST=3
Status: Bus=2 Dev=1 Func=0 64bit+ 133MHz+ SCD- USC-,
DC=simple, DMMRBC=0, DMOST=3, DMCRS=0, RSCEM-

Quoted 7 lines · Mirko Benz

Hello,

We have created a RAID 0 for the same environment:
Personalities : [raid0] [raid5]
md0 : active raid0 sdi[7] sdh[6] sdg[5] sdf[4] sde[3] sdd[2] sdc[1] sdb[0]

      1250326528 blocks 64k chunks

Personalities : [linear] [raid0] [raid1] [raid5] [multipath] [raid6] [raid10] [faulty]
md0 : active raid0 sdh[7] sdg[6] sdf[5] sde[4] sdd[3] sdc[2] sdb[1] sda
[0]

      3125690368 blocks 64k chunks

SCSI disks: 400GB SATA

host: scsi11 Channel: 00 Id: 00 Lun: 00
Vendor: Hitachi Model: HDS724040KLSA80 Rev: KFAO
Type: Direct-Access ANSI SCSI revision: 03

Quoted 34 lines

*** dd TEST ***

time dd if=/dev/zero of=/dev/md0 bs=1M
14967373824 bytes transferred in 32,060497 seconds (466847843 bytes/sec)

iostat 5 output:
avg-cpu: %user %nice %sys %iowait %idle

           0,00    0,00   89,60    9,50    0,90

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn

hda               0,00         0,00         0,00          0          0
sda               0,00         0,00         0,00          0          0
sdb             455,31         0,00    116559,52          0     581632
sdc             455,51         0,00    116540,28          0     581536
sdd             450,10         0,00    116545,09          0     581560
sde             454,11         0,00    116559,52          0     581632
sdf             452,30         0,00    116559,52          0     581632
sdg             454,71         0,00    116553,11          0     581600
sdh             453,31         0,00    116533,87          0     581504
sdi             453,91         0,00    116556,31          0     581616
sdj               0,00         0,00         0,00          0          0
sdk               0,00         0,00         0,00          0          0
sdl               0,00         0,00         0,00          0          0
sdm               0,00         0,00         0,00          0          0
sdn               0,00         0,00         0,00          0          0

md0 116556,11 0,00 932448,90 0 4652920

Comments: 466 MB / 8 = 58,25 MB/s which is about the same as a dd to a single disk (58,5 MB/s). So the controller + I/O subsystem is not the bottleneck.

Regards,
Mirko

#9 lore raw
In reply to Mirko Benz

i would like to suggest u to do a 4+1 raid5 configuration and see what happen.

Ming

Quoted 41 lines · Mirko Benz

Hello,

We have created a RAID 0 for the same environment:
Personalities : [raid0] [raid5]
md0 : active raid0 sdi[7] sdh[6] sdg[5] sdf[4] sde[3] sdd[2] sdc[1] sdb[0]

      1250326528 blocks 64k chunks

*** dd TEST ***

time dd if=/dev/zero of=/dev/md0 bs=1M
14967373824 bytes transferred in 32,060497 seconds (466847843 bytes/sec)

iostat 5 output:
avg-cpu: %user %nice %sys %iowait %idle

           0,00    0,00   89,60    9,50    0,90

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn

hda               0,00         0,00         0,00          0          0
sda               0,00         0,00         0,00          0          0
sdb             455,31         0,00    116559,52          0     581632
sdc             455,51         0,00    116540,28          0     581536
sdd             450,10         0,00    116545,09          0     581560
sde             454,11         0,00    116559,52          0     581632
sdf             452,30         0,00    116559,52          0     581632
sdg             454,71         0,00    116553,11          0     581600
sdh             453,31         0,00    116533,87          0     581504
sdi             453,91         0,00    116556,31          0     581616
sdj               0,00         0,00         0,00          0          0
sdk               0,00         0,00         0,00          0          0
sdl               0,00         0,00         0,00          0          0
sdm               0,00         0,00         0,00          0          0
sdn               0,00         0,00         0,00          0          0

md0 116556,11 0,00 932448,90 0 4652920

Comments: 466 MB / 8 = 58,25 MB/s which is about the same as a dd to a single disk (58,5 MB/s). So the controller + I/O subsystem is not the bottleneck.

Regards,
Mirko

author
#10 lore raw

Hello,

Here are the results using 5 disks for RAID 5 – basically the same results but with lower values. Again, much slower than it could be.

*** dd TEST ***

time dd if=/dev/zero of=/dev/md0 bs=1M
2819620864 bytes transferred in 23,410720 seconds (120441442 bytes/sec)

iostat 5 output:
avg-cpu: %user %nice %sys %iowait %idle
0,10 0,00 55,40 36,20 8,30

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn hda 0,00 0,00 0,00 0 0
sda 0,00 0,00 0,00 0 0
sdb 345,09 1999,20 71501,40 9976 356792
sdc 348,10 2412,83 71434,07 12040 356456 sdd 356,31 2460,92 71748,30 12280 358024 sde 351,50 2456,11 71058,92 12256 354584 sdf 348,10 2008,82 70935,47 10024 353968 sdg 0,00 0,00 0,00 0 0
sdh 0,00 0,00 0,00 0 0
sdi 0,00 0,00 0,00 0 0
sdj 0,00 0,00 0,00 0 0
sdk 0,00 0,00 0,00 0 0
sdl 0,00 0,00 0,00 0 0
sdm 0,00 0,00 0,00 0 0
sdn 0,00 0,00 0,00 0 0
md0 35226,65 0,00 281813,23 0 1406248

disktest gives 99 MB/s but shows the same behaviour (unbalanced usage of disks).

Regards,
Mirko

Quoted 53 lines · Ming Zhang

i would like to suggest u to do a 4+1 raid5 configuration and see what happen.

Ming

Quoted 41 lines · Mirko Benz

Hello,

We have created a RAID 0 for the same environment:
Personalities : [raid0] [raid5]
md0 : active raid0 sdi[7] sdh[6] sdg[5] sdf[4] sde[3] sdd[2] sdc[1] sdb[0]

     1250326528 blocks 64k chunks

*** dd TEST ***

time dd if=/dev/zero of=/dev/md0 bs=1M
14967373824 bytes transferred in 32,060497 seconds (466847843 bytes/sec)

iostat 5 output:

avg-cpu:  %user   %nice    %sys %iowait   %idle
          0,00    0,00   89,60    9,50    0,90

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn

hda               0,00         0,00         0,00          0          0
sda               0,00         0,00         0,00          0          0
sdb             455,31         0,00    116559,52          0     581632
sdc             455,51         0,00    116540,28          0     581536
sdd             450,10         0,00    116545,09          0     581560
sde             454,11         0,00    116559,52          0     581632
sdf             452,30         0,00    116559,52          0     581632
sdg             454,71         0,00    116553,11          0     581600
sdh             453,31         0,00    116533,87          0     581504
sdi             453,91         0,00    116556,31          0     581616
sdj               0,00         0,00         0,00          0          0
sdk               0,00         0,00         0,00          0          0
sdl               0,00         0,00         0,00          0          0
sdm               0,00         0,00         0,00          0          0
sdn               0,00         0,00         0,00          0          0

md0 116556,11 0,00 932448,90 0 4652920

Comments: 466 MB / 8 = 58,25 MB/s which is about the same as a dd to a single disk (58,5 MB/s). So the controller + I/O subsystem is not the bottleneck.

Regards,
Mirko

-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#11 lore raw

i think this is out of my mind now. i think 4+1 DISK with 64KB chunk size is 256KB which fit the 1M size well. So there should not have any read happen...

ming

Quoted 33 lines · Mirko Benz

Hello,

Here are the results using 5 disks for RAID 5 – basically the same results but with lower values.
Again, much slower than it could be.

*** dd TEST ***

time dd if=/dev/zero of=/dev/md0 bs=1M
2819620864 bytes transferred in 23,410720 seconds (120441442 bytes/sec)

iostat 5 output:
avg-cpu: %user %nice %sys %iowait %idle
0,10 0,00 55,40 36,20 8,30

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn hda 0,00 0,00 0,00 0 0
sda 0,00 0,00 0,00 0 0
sdb 345,09 1999,20 71501,40 9976 356792
sdc 348,10 2412,83 71434,07 12040 356456 sdd 356,31 2460,92 71748,30 12280 358024 sde 351,50 2456,11 71058,92 12256 354584 sdf 348,10 2008,82 70935,47 10024 353968 sdg 0,00 0,00 0,00 0 0
sdh 0,00 0,00 0,00 0 0
sdi 0,00 0,00 0,00 0 0
sdj 0,00 0,00 0,00 0 0
sdk 0,00 0,00 0,00 0 0
sdl 0,00 0,00 0,00 0 0
sdm 0,00 0,00 0,00 0 0
sdn 0,00 0,00 0,00 0 0
md0 35226,65 0,00 281813,23 0 1406248

Quoted 64 lines

disktest gives 99 MB/s but shows the same behaviour (unbalanced usage of disks).

Regards,
Mirko

Quoted 55 lines · Ming Zhang

i would like to suggest u to do a 4+1 raid5 configuration and see what happen.

Ming

Quoted 43 lines · Mirko Benz

Hello,

We have created a RAID 0 for the same environment:
Personalities : [raid0] [raid5]
md0 : active raid0 sdi[7] sdh[6] sdg[5] sdf[4] sde[3] sdd[2] sdc[1] sdb[0]

     1250326528 blocks 64k chunks

*** dd TEST ***

time dd if=/dev/zero of=/dev/md0 bs=1M
14967373824 bytes transferred in 32,060497 seconds (466847843 bytes/sec)

iostat 5 output:

avg-cpu:  %user   %nice    %sys %iowait   %idle
          0,00    0,00   89,60    9,50    0,90

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn

hda               0,00         0,00         0,00          0          0
sda               0,00         0,00         0,00          0          0
sdb             455,31         0,00    116559,52          0     581632
sdc             455,51         0,00    116540,28          0     581536
sdd             450,10         0,00    116545,09          0     581560
sde             454,11         0,00    116559,52          0     581632
sdf             452,30         0,00    116559,52          0     581632
sdg             454,71         0,00    116553,11          0     581600
sdh             453,31         0,00    116533,87          0     581504
sdi             453,91         0,00    116556,31          0     581616
sdj               0,00         0,00         0,00          0          0
sdk               0,00         0,00         0,00          0          0
sdl               0,00         0,00         0,00          0          0
sdm               0,00         0,00         0,00          0          0
sdn               0,00         0,00         0,00          0          0

md0 116556,11 0,00 932448,90 0 4652920

Comments: 466 MB / 8 = 58,25 MB/s which is about the same as a dd to a single disk (58,5 MB/s). So the controller + I/O subsystem is not the bottleneck.

Regards,
Mirko

-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#12 lore raw
In reply to Mirko Benz
Quoted 5 lines · On Friday August 26, [email protected] wrote:

Hello,

Here are the results using 5 disks for RAID 5 $,1rs(B basically the same results but with lower values.
Again, much slower than it could be.

Yes, there definitely does seem to be something wrong with raid5 write speed, doesn't there!!

I will try to look into it over the next couple of weeks.

NeilBrown

#13 lore raw
In reply to Mirko Benz

----- Original Message -----
From: "Mirko Benz" <[email protected]>
To: "Neil Brown" <[email protected]>
Cc: <[email protected]>; "Linux RAID" <[email protected]>
Sent: Thursday, August 25, 2005 6:38 PM
Subject: Re: RAID 5 write performance advice

Quoted 12 lines

Hello,

We intend to export a lvm/md volume via iSCSI or SRP using InfiniBand to remote clients. There is no local file system processing on the storage platform. The clients may have a variety of file systems including ext3, GFS.

Single disk write performance is: 58,5 MB/s. With large sequential write operations I would expect something like 90% of n-1 * single_disk_performance if stripe write can be utilized. So roughly 400 MB/s – which the HW RAID devices achieve.

Quoted 7 lines

RAID setup:
Personalities : [raid0] [raid5]
md0 : active raid5 sdi[7] sdh[6] sdg[5] sdf[4] sde[3] sdd[2] sdc[1] sdb[0] 1094035712 blocks level 5, 64k chunk, algorithm 2 [8/8] [UUUUUUUU]

We have assigned the deadline scheduler to every disk in the RAID. The default scheduler gives much lower results.

I recommend you to try not just another iosched, try they settings too! For me, this helps... ;-)

(the settings is in the kernel's Documentation/block -dir)

Additionally try to set sched's, in another layer, eg: lvm.... (if it is possible, i don't know...)

I use 8TB disk in one big raid0 array, and my config is this:
4 PC each 11x200GB hdd with RAID5, exports the four 2TB space. In the clients I use the default anticipatory sched with this settings:
antic_expire : 6
read_batch_expire: 500
read_expire 125
write_batch_expire 125
write_expire 250

And one dual Xeon system, the TOP-client inside the 8TB raid0 (from 4 disk node).
I use GNBD, and for the gnbd devices, the deadline sched is the best, with this settings:
fifo_batch: 16
front_merges 0
read_expire 50
write_expire 5000
writes_starved 255 ( - 1024 depends on what I want....)

I try the LVM too, but I dropped it, because too low performace... :( I try to grow with raid's linear mode insted. ;-)

Thanks to Neilbrown! ;-)

( I didn't test the patch yet)

Quoted 65 lines

*** dd TEST ***

time dd if=/dev/zero of=/dev/md0 bs=1M
5329911808 bytes transferred in 28,086199 seconds (189769779 bytes/sec)

iostat 5 output:
avg-cpu: %user %nice %sys %iowait %idle
0,10 0,00 87,80 7,30 4,80

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn hda 0,00 0,00 0,00 0 0
sda 0,00 0,00 0,00 0 0
sdb 1976,10 1576,10 53150,60 7912 266816 sdc 2072,31 1478,88 53150,60 7424 266816 sdd 2034,06 1525,10 53150,60 7656 266816 sde 1988,05 1439,04 53147,41 7224 266800 sdf 1975,10 1499,60 53147,41 7528 266800 sdg 1383,07 1485,26 53145,82 7456 266792 sdh 1562,55 1311,55 53145,82 6584 266792 sdi 1586,85 1295,62 53145,82 6504 266792 sdj 0,00 0,00 0,00 0 0
sdk 0,00 0,00 0,00 0 0
sdl 0,00 0,00 0,00 0 0
sdm 0,00 0,00 0,00 0 0
sdn 0,00 0,00 0,00 0 0
md0 46515,54 0,00 372124,30 0 1868064

Comments: Large write should not see any read operations. But there are some???

*** disktest ***

disktest -w -PT -T30 -h1 -K8 -B512k -ID /dev/md0

| 2005/08/25-17:27:04 | STAT | 4072 | v1.1.12 | /dev/md0 | Write
throughput: 160152507.7B/s (152.73MB/s), IOPS 305.7/s. | 2005/08/25-17:27:05 | STAT | 4072 | v1.1.12 | /dev/md0 | Write
throughput: 160694272.0B/s (153.25MB/s), IOPS 306.6/s. | 2005/08/25-17:27:06 | STAT | 4072 | v1.1.12 | /dev/md0 | Write
throughput: 160339606.6B/s (152.91MB/s), IOPS 305.8/s.

iostat 5 output:
avg-cpu: %user %nice %sys %iowait %idle
38,96 0,00 50,25 5,29 5,49

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn hda 0,00 0,00 0,00 0 0
sda 1,20 0,00 11,18 0 56
sdb 986,43 0,00 39702,99 0 198912
sdc 922,75 0,00 39728,54 0 199040
sdd 895,81 0,00 39728,54 0 199040
sde 880,84 0,00 39728,54 0 199040
sdf 839,92 0,00 39728,54 0 199040
sdg 842,91 0,00 39728,54 0 199040
sdh 1557,49 0,00 79431,54 0 397952
sdi 2246,71 0,00 104411,98 0 523104
sdj 0,00 0,00 0,00 0 0
sdk 0,00 0,00 0,00 0 0
sdl 0,00 0,00 0,00 0 0
sdm 0,00 0,00 0,00 0 0
sdn 0,00 0,00 0,00 0 0
md0 1550,70 0,00 317574,45 0 1591048

Quoted 3 lines

Comments:
Zero read requests – as it should be. But the write requests are not proportional. sdh and sdi have significantly more requests???

I have get this too in 2.6.13-rc3, but it is gone in rc6, and 2.6.13! What version of kernel do you use?

Janos

Quoted 7 lines

The write requests to the disks of the RAID should be 1/7 higher than to the md device.
But there are significantly more write operations.

All these operations are to the raw device. Setting up a ext3 fs we get around 127 MB/s with dd.

Quoted 8 lines

Any idea?

--Mirko

-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html