Skip to contentSkip to conversation

striping of a 4 drive raid10

DISCUSSION Keld Jørn Simonsen 4 participants 8 messages Jan 27, 2008 – Jan 28, 2008 2 messages not in the archive MD RAID

lore
Share & tools
lore
#0 lore raw

Hi

I have tried to make a striping raid out of my new 4 x 1 TB SATA-2 disks. I tried raid10,f2 in several ways:

1: md0 = raid10,f2 of sda1+sdb1, md1= raid10,f2 of sdc1+sdd1, md2 = raid0 of md0+md1

2: md0 = raid0 of sda1+sdb1, md1= raid0 of sdc1+sdd1, md2 = raid01,f2 of md0+md1

3: md0 = raid10,f2 of sda1+sdb1, md1= raid10,f2 of sdc1+sdd1, chunksize of md0 =md1 =128 KB, md2 = raid0 of md0+md1 chunksize = 256 KB

4: md0 = raid0 of sda1+sdb1, md1= raid0 of sdc1+sdd1, chunksize of md0 = md1 = 128 KB, md2 = raid01,f2 of md0+md1 chunksize = 256 KB

5: md0= raid10,f4 of sda1+sdb1+sdc1+sdd1

My new disks give a transfer rate of about 80 MB/s, so I expected to have something like 320 MB/s for the whole raid, but I did not get more than about 180 MB/s.

I think it may be something with the layout, that in effect the drives should be something like:

sda1 sdb1 sdc1 sdd1
0 1 2 3
4 5 6 7

And this was not really doable for the combination of raids, because thet combinations give different block layouts.

How can it be done? Do we need a new raid type?

Best regards
keld

#1 lore raw
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

On Sun, 27 Jan 2008 20:33:45 +0100, Keld Jørn Simonsen <[email protected]> said:

keld> Hi I have tried to make a striping raid out of my new 4 x keld> 1 TB SATA-2 disks. I tried raid10,f2 in several ways:

keld> 1: md0 = raid10,f2 of sda1+sdb1, md1= raid10,f2 of sdc1+sdd1, md2 = raid0
keld> of md0+md1
keld> 2: md0 = raid0 of sda1+sdb1, md1= raid0 of sdc1+sdd1, md2 = raid01,f2
keld> of md0+md1
keld> 3: md0 = raid10,f2 of sda1+sdb1, md1= raid10,f2 of sdc1+sdd1, chunksize of
keld> md0 =md1 =128 KB, md2 = raid0 of md0+md1 chunksize = 256 KB
keld> 4: md0 = raid0 of sda1+sdb1, md1= raid0 of sdc1+sdd1, chunksize
keld> of md0 = md1 = 128 KB, md2 = raid01,f2 of md0+md1 chunksize = 256 KB

These stacked RAID levels don't make a lot of sense.

keld> 5: md0= raid10,f4 of sda1+sdb1+sdc1+sdd1

This also does not make a lot of sense. Why have four mirrors instead of two?

Instead, try 'md0 = raid10,f2' for example. The first mirror of will be striped across the outer half of all four drives, and the second mirrors will be rotated in the inner half of each drive.

Which of course means that reads will be quite quick, but writes and degraded operation will be slower.

Consider this post for more details:

http://www.spinics.net/lists/raid/msg18130.html

[ ... ]
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#3 lore raw
In reply to Keld Jørn Simonsen
Quoted 11 lines · On Sunday January 27, [email protected] wrote:

Hi

I have tried to make a striping raid out of my new 4 x 1 TB SATA-2 disks. I tried raid10,f2 in several ways:

1: md0 = raid10,f2 of sda1+sdb1, md1= raid10,f2 of sdc1+sdd1, md2 = raid0 of md0+md1

2: md0 = raid0 of sda1+sdb1, md1= raid0 of sdc1+sdd1, md2 = raid01,f2 of md0+md1

Quoted 7 lines

3: md0 = raid10,f2 of sda1+sdb1, md1= raid10,f2 of sdc1+sdd1, chunksize of md0 =md1 =128 KB, md2 = raid0 of md0+md1 chunksize = 256 KB

4: md0 = raid0 of sda1+sdb1, md1= raid0 of sdc1+sdd1, chunksize of md0 = md1 = 128 KB, md2 = raid01,f2 of md0+md1 chunksize = 256 KB

5: md0= raid10,f4 of sda1+sdb1+sdc1+sdd1

Try
6: md0 = raid10,f2 of sda1+sdb1+sdc1+sdd1

Also try raid10,o2 with a largeish chunksize (256KB is probably big enough).

NeilBrown

Quoted 23 lines

My new disks give a transfer rate of about 80 MB/s, so I expected to have something like 320 MB/s for the whole raid, but I did not get more than about 180 MB/s.

I think it may be something with the layout, that in effect the drives should be something like:

sda1 sdb1 sdc1 sdd1
0 1 2 3
4 5 6 7

And this was not really doable for the combination of raids, because thet combinations give different block layouts.

How can it be done? Do we need a new raid type?

Best regards
keld
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#7 lore raw
In reply to a message not in the archive
Quoted 15 lines · Neil Brown
Quoted 14 lines · On Sunday January 27, [email protected] wrote:

Hi

I have tried to make a striping raid out of my new 4 x 1 TB SATA-2 disks. I tried raid10,f2 in several ways:

1: md0 = raid10,f2 of sda1+sdb1, md1= raid10,f2 of sdc1+sdd1, md2 = raid0 of md0+md1

2: md0 = raid0 of sda1+sdb1, md1= raid0 of sdc1+sdd1, md2 = raid01,f2 of md0+md1

3: md0 = raid10,f2 of sda1+sdb1, md1= raid10,f2 of sdc1+sdd1, chunksize of md0 =md1 =128 KB, md2 = raid0 of md0+md1 chunksize = 256 KB

Quoted 7 lines
Quoted 4 lines

4: md0 = raid0 of sda1+sdb1, md1= raid0 of sdc1+sdd1, chunksize of md0 = md1 = 128 KB, md2 = raid01,f2 of md0+md1 chunksize = 256 KB

5: md0= raid10,f4 of sda1+sdb1+sdc1+sdd1

Try
6: md0 = raid10,f2 of sda1+sdb1+sdc1+sdd1

That I already tried, (and I wrongly stated that I used f4 in stead of f2). I had two times a thruput of about 300 MB/s but since then I could not reproduce the behaviour. Are there errors on this that has been corrected in newer kernels?

Quoted 2 lines

Also try raid10,o2 with a largeish chunksize (256KB is probably big enough).

I tried that too, but my mdadm did not allow me to use the o flag.

My kernel is 2.6.12 and mdadm is v1.12.0 - 14 June 2005. can I upgrade the mdadm alone to a newer version, and then which is recommendable?

best regards
keld

#2 lore raw
In reply to Peter Grandi
Quoted 15 lines · Peter Grandi
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

On Sun, 27 Jan 2008 20:33:45 +0100, Keld Jørn Simonsen <[email protected]> said:

keld> Hi I have tried to make a striping raid out of my new 4 x keld> 1 TB SATA-2 disks. I tried raid10,f2 in several ways:

keld> 1: md0 = raid10,f2 of sda1+sdb1, md1= raid10,f2 of sdc1+sdd1, md2 = raid0
keld> of md0+md1
keld> 2: md0 = raid0 of sda1+sdb1, md1= raid0 of sdc1+sdd1, md2 = raid01,f2
keld> of md0+md1
keld> 3: md0 = raid10,f2 of sda1+sdb1, md1= raid10,f2 of sdc1+sdd1, chunksize of
keld> md0 =md1 =128 KB, md2 = raid0 of md0+md1 chunksize = 256 KB
keld> 4: md0 = raid0 of sda1+sdb1, md1= raid0 of sdc1+sdd1, chunksize
keld> of md0 = md1 = 128 KB, md2 = raid01,f2 of md0+md1 chunksize = 256 KB

Quoted 6 lines

These stacked RAID levels don't make a lot of sense.

keld> 5: md0= raid10,f4 of sda1+sdb1+sdc1+sdd1

This also does not make a lot of sense. Why have four mirrors instead of two?

My error, I did mean f2.

Anyway 4 mirrors would make the disk 2 times faster than 2 disks, and given disk prices these days this could make a lot of sense.

Quoted 11 lines

Instead, try 'md0 = raid10,f2' for example. The first mirror of will be striped across the outer half of all four drives, and the second mirrors will be rotated in the inner half of each drive.

Which of course means that reads will be quite quick, but writes and degraded operation will be slower.

Consider this post for more details:

http://www.spinics.net/lists/raid/msg18130.html

Thanks for the reference.

There is also more in the original article on possible layouts of what is now known as raid10,f2

http://marc.info/?l=linux-raid&m=107427614604701&w=2

including performance enhancements due to use of the faster outer sectors, and smaller average seek times because you can seek on only half the disk.

best regards
keld
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#8 lore raw
In reply to Keld Jørn Simonsen
Quoted 21 lines · On Sunday January 27, [email protected] wrote:
Quoted 20 lines · Neil Brown
Quoted 18 lines · On Sunday January 27, [email protected] wrote:

Hi

I have tried to make a striping raid out of my new 4 x 1 TB SATA-2 disks. I tried raid10,f2 in several ways:

1: md0 = raid10,f2 of sda1+sdb1, md1= raid10,f2 of sdc1+sdd1, md2 = raid0 of md0+md1

2: md0 = raid0 of sda1+sdb1, md1= raid0 of sdc1+sdd1, md2 = raid01,f2 of md0+md1

3: md0 = raid10,f2 of sda1+sdb1, md1= raid10,f2 of sdc1+sdd1, chunksize of md0 =md1 =128 KB, md2 = raid0 of md0+md1 chunksize = 256 KB

4: md0 = raid0 of sda1+sdb1, md1= raid0 of sdc1+sdd1, chunksize of md0 = md1 = 128 KB, md2 = raid01,f2 of md0+md1 chunksize = 256 KB

5: md0= raid10,f4 of sda1+sdb1+sdc1+sdd1

Quoted 7 lines
Quoted 2 lines

Try
6: md0 = raid10,f2 of sda1+sdb1+sdc1+sdd1

That I already tried, (and I wrongly stated that I used f4 in stead of f2). I had two times a thruput of about 300 MB/s but since then I could not reproduce the behaviour. Are there errors on this that has been corrected in newer kernels?

No, I don't think any performance related changes have been made to raid10 lately.

You could try increasing the read-ahead size. For a 4-drive raid10 it defaults to 4 times the read-ahead setting of a single drive, but increasing substantially (e.g. 64 times) seem to increase the speed of "dd" reading a gigabyte.
Whether that will actually affect your target workload is a different question.

Quoted 10 lines
Quoted 4 lines

Also try raid10,o2 with a largeish chunksize (256KB is probably big enough).

I tried that too, but my mdadm did not allow me to use the o flag.

My kernel is 2.6.12 and mdadm is v1.12.0 - 14 June 2005. can I upgrade the mdadm alone to a newer version, and then which is recommendable?

You would need a newer kernel and a newer mdadm to get raid10 - offset mode.

NeilBrown

#9 lore raw
In reply to Keld Jørn Simonsen
Quoted 37 lines · Keld Jørn Simonsen
Quoted 15 lines · Neil Brown
Quoted 13 lines · On Sunday January 27, [email protected] wrote:

Hi

I have tried to make a striping raid out of my new 4 x 1 TB SATA-2 disks. I tried raid10,f2 in several ways:

1: md0 = raid10,f2 of sda1+sdb1, md1= raid10,f2 of sdc1+sdd1, md2 = raid0 of md0+md1

2: md0 = raid0 of sda1+sdb1, md1= raid0 of sdc1+sdd1, md2 = raid01,f2 of md0+md1

3: md0 = raid10,f2 of sda1+sdb1, md1= raid10,f2 of sdc1+sdd1, chunksize of md0 =md1 =128 KB, md2 = raid0 of md0+md1 chunksize = 256 KB

Quoted 8 lines
Quoted 5 lines

4: md0 = raid0 of sda1+sdb1, md1= raid0 of sdc1+sdd1, chunksize of md0 = md1 = 128 KB, md2 = raid01,f2 of md0+md1 chunksize = 256 KB

5: md0= raid10,f4 of sda1+sdb1+sdc1+sdd1

Try
6: md0 = raid10,f2 of sda1+sdb1+sdc1+sdd1

That I already tried, (and I wrongly stated that I used f4 in stead of f2). I had two times a thruput of about 300 MB/s but since then I could not reproduce the behaviour. Are there errors on this that has been corrected in newer kernels?

Quoted 3 lines

Also try raid10,o2 with a largeish chunksize (256KB is probably big enough).

Quoted 6 lines

I tried that too, but my mdadm did not allow me to use the o flag.

My kernel is 2.6.12 and mdadm is v1.12.0 - 14 June 2005. can I upgrade the mdadm alone to a newer version, and then which is recommendable?

I doubt that updating the mdadm is going to help, the kernel is old and lacks a number of improvements in the last few years. I don't think you will see any major improvements without a kernel upgrade.

Signature · 10 lines
Bill Davidsen <[email protected]>
 "Woe unto the statesman who makes war without a reason that will still
 be valid when the war is over..." Otto von Bismark 



-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#5 lore raw
In reply to a message not in the archive
Quoted 26 lines · Bill Davidsen
Quoted 25 lines · Neil Brown
Quoted 11 lines · On Sunday January 27, [email protected] wrote:

Hi

I have tried to make a striping raid out of my new 4 x 1 TB SATA-2 disks. I tried raid10,f2 in several ways:

1: md0 = raid10,f2 of sda1+sdb1, md1= raid10,f2 of sdc1+sdd1, md2 = raid0 of md0+md1

2: md0 = raid0 of sda1+sdb1, md1= raid0 of sdc1+sdd1, md2 = raid01,f2 of md0+md1

Quoted 8 lines

3: md0 = raid10,f2 of sda1+sdb1, md1= raid10,f2 of sdc1+sdd1, chunksize of md0 =md1 =128 KB, md2 = raid0 of md0+md1 chunksize = 256 KB

4: md0 = raid0 of sda1+sdb1, md1= raid0 of sdc1+sdd1, chunksize of md0 = md1 = 128 KB, md2 = raid01,f2 of md0+md1 chunksize = 256 KB

5: md0= raid10,f4 of sda1+sdb1+sdc1+sdd1

Try
6: md0 = raid10,f2 of sda1+sdb1+sdc1+sdd1

Quoted 8 lines
Quoted 3 lines

Also try raid10,o2 with a largeish chunksize (256KB is probably big enough).

Looking at the issues raised, there might be some benefit from having the mirror chunks on the slower inner tracks of a raid10, and to read from the outer tracks if the drives with the data on the outer tracks are idle. This would appear to offer a transfer rate benefit overall.

Hmm, how do I do this? I think this is normal behaviour of a raid10,f2. Is that so?

So you mean I should rather use f2 than o2? Or should I configure the f2 in some way?

My hdparm -t gives:

/dev/sda5:

 Timing buffered beginning disk reads:   82 MB in  1.00 seconds = 81.686 MB/sec
 Timing buffered ending    disk reads:   42 MB in  1.03 seconds = 40.625 MB/sec

Average seek time 13.714 msec, min=4.641, max=23.921 Average track-to-track time 28.151 msec, min=26.729, max=28.730

So, yes, there is a reason to use the faster outer tracks - and have the faster access time that f2 gives . How does o2 behave here? Does it read and search on the whole disk?

As to your other comments in another mail, I could of cause install a newer kernel and mdadm, but then I would loose the support of my supported and paid system. And Neil said that there have been no performance fixes for f2 since the kernel I use (2.6.12).
I thought that o2 support was included since 2.6.10 - but apparantly not so.

Best regards
keld