Help with chunksize on raid10 -p o3 array

Hi,
I have been trying to figure out the best chunk size for raid10 before migrating my server to it (currently raid1). I am looking at 3 offset stripes, as I want to have two drive failure redundancy, and offset striping is said to have the best write performance, with read performance equal to far. Information on the internet is scarce so I decided to test chunking myself. I used the script http://rabbit.us/pool/misc/raid_test.txt to iterate through different chunk sizes, and try to dd the resulting array to /dev/null. I deliberately did not make a filesystem on top of the array - I was just looking for raw performance, and since the FS layer is not involved no caching/optimization is taking place. I also monitored the process with dstat in a separate window, and memory usage confirmed that this method is valid. I got some pretty weird results: http://rabbit.us/pool/misc/raid_test_results.txt
From all my readings so far I thought that with chunk size increase the large block access throughput decreases while small block reads increase, and it is just a matter of finding a "sweet spot" balancing them out. The results, however, clearly show something else. There are some inconsistencies, which I attribute to my non-scientific approach, but the trend is clearly showing.

Here are the questions I have:

* Why did the test show best consistent performance over a 16k chunk? Is there a way to determine this number without running a lengthy benchmark, just from knowing the drive performance?

* Why although I have 3 identical chunks of data at any time, dstat never showed simultaneous reading from more than 2 drives. Every dd run was accompanied by maxing out one of the drives at 58MB/s and another one was trying to catch up to various degrees depending on the chunk size. Then on the next dd run two other drives would be (seemingly random) selected and the process would repeat.

* What the test results don't show but dstat did is how the array resync behaved after the array creation. Although my system can sustain reads from all 4 drives at the max speed of 58MB/s, here is what the resync at different chunk sizes looked like:

32k - simultaneous reads from all 4 drives at 47MB/s sustained
64k - simultaneous reads from all 4 drives at 56MB/s sustained
128k - simultaneous reads from all 4 drives at 54MB/s sustained
512k - simultaneous reads from all 4 drives at 30MB/s sustained
1024k - simultaneous reads from all 4 drives at 38MB/s sustained
4096k - simultaneous reads from all 4 drives at 44MB/s sustained
16384k - simultaneous reads from all 4 drives at 46MB/s sustained
32768k - simultaneous reads from 2 drives at 58MB/s sustained and
the other two at 26MB/s sustained alternating the speed between the pairs of drives every 3 seconds or so
65536k - All 4 drives started at 58MB/s sustained gradually
reducing to 44MB/s sustained at the same time

I repeated just the creation of arrays - the results are consistent. Is there any explanation for this?

By any chance did you remember to increase stripe_cache_size to match the chunk size? If not, there you go.

bill davidsen <[email protected]>
 CTO TMR Associates, Inc
 Doing interesting things with small computers since 1979

At the end of /usr/src/linux/Documentation/md.txt it specifically says that stripe_cache_size is raid5 specific, and it made sense to me, as caching stuff to avoid re-doing parity is beneficial. I will test later today trying to set the cache higher. Are there any guidelines on how large should it be in relation to the chunk size/number of drives for raid10?

The different block sizes in the reads will make very little difference to the results as the kernel will be doing read-ahead for you. If you want to really test throughput at different block sizes you need to insert random seeks.

When you are doing a large sequential read from a raid10-offset array, it will (should) read from all drives for the first chunk, then skip over the offset-copy(s) of that chunk and read again. Thus you get a read-seek-read-seek pattern.
The time that each read takes will be proportional to the chunk size. The time that the seek take will be pretty stable (mostly head settling time I believe). So for small chunks, the seek time dominates. For large chunks, the read starts to dominate.

I think this is what you are seeing. When you get to 16M chunks (16K kilobytes) the seek time is a smaller fraction of the read time and so you spend more time reading.

The fact that it drops off again at 32M is probably due to some limit in the amount of read-ahead that the kernel will initiate. If it won't issue the request to the last drive before the request to the first drive completes, you will obviously get slower throughput.

Poor read-balancing code. It really needs more thought. Possibly for raid10 we shouldn't try to balance at all. Just read from the 'first' copy in each case....

A raid10 resync involves reading all copies of each block and doing comparisons. If we find a difference, we write out one copy over the rest.

We issue the requests in sequential order for the blocks. If you think about how the blocks are laid out, you will see that this is not always sequential order on each individual device. In some cases we will ask to read a later device block before an earlier device block.

For small chunk sizes, the amount of backward-seeking will be fairly small and the elevator will probably absorb all of it. For larger chunk sizes, you will get longer backward seeking that doesn't get rearranged by the elevator and so you will get lower throughout.

Exactly where the interesting 32M artifact comes from I don't know. It could relate to the window size used by md - there is a limit to how many outstanding resync requests there can be at one time. It limits to 32 requests each 64K is size which multiplies out to 2Meg..... not obvious how that connects, is it?

NeilBrown

Neil, thank you for the time and effort to answer my previous email. Excellent insights. I thought that read-ahead is filesystem specific, and subsequently I would be safe to use the raw device. I will definitely test with bonnie again.

Is this anywhere near the top of the todo list, or for now raid10 users are bound to a maximum read speed of a two drive combination?

And a last question - earlier in this thread Bill Davidsen suggested to play with the stripe_cache_size. I tried to increase it (did just two tests though) with no apparent effect. Does this setting apply to raid1/10 at all or it is strictly in the raid5/6 domain? If so, are there any tweaks apart from the chunk size and the layout that can affect raid10 performance?

Once again thank you for the help.

Peter

I have not done any testing with the md native RAID10 implementations, so perhaps there are some other advantages, but have you tried setting up your 4 drives as a RAID 0 made up of a pair of RAID1s?

I have had very good performance in this configuration - reads and writes of over 140MB/s for 4 x 7200 SATA.

Regards,

Richard

The advantage is higher redundancy when I can have any two drives fail in a x3 layout unlike the raid1/0 setup, although I sacrifice available disk space. But yes, I agree that if I was after pure throughput raid1/0 would be more beneficial, with the downside of 1.5 disk failure redundancy.

Incorporating suggestions from previous posts (thank you everyone), I used this modified script at http://rabbit.us/pool/misc/raid_test2.txt To negate effects of caching memory was jammed below 200mb free by using a full tmpfs mount with no swap. Here is what I got with far layout (-p f3): http://rabbit.us/pool/misc/raid_far.html The clear winner is 1M chunks, and is very consistent at any block size. I was surprised even more to see that my read speed was identical to that of a raid0 getting near the _maximum_ physical speed of 4 drives (roughly 55MB sustained across 1.2G). Unlike offset layout, far really shines at reading stuff back. The write speed did not suffer noticeably compared to offset striping. Here are the results (-p o3) for comparison: http://rabbit.us/pool/misc/raid_offset.html, and they roughly seem to correlate with my earlier testing using dd.

So I guess the way to go for this system will be f3, although the md(4) says that offset layout should be more beneficial. Is there anything I missed while setting my o3 array, so that I got worse performance for both read and write compared to f3?

Once again thanks everyone for the help.
Peter