Skip to contentSkip to conversation

RAID-6

DISCUSSION H. Peter Anvin 6 participants 16 messages Nov 11, 2002 – Nov 13, 2002 MD RAID

lore
Share & tools
lore
#0 lore raw

Hi all,

I'm playing around with RAID-6 algorithms lately. With RAID-6 I mean a setup which needs N+2 disks for N disks worth of storage and can handle any two disks failing -- this seems to be the contemporary definition of RAID-6 (the originally proposed "two-dimensional parity" which required N+2*sqrt(N) drives never took off for obvious reasons.)

Based on my current research, I think the following should be true:

a) write performance will be worse than RAID-5, but I believe it can be kept to within a factor of 1.5-2.0 on machines with suitable SIMD instruction sets (e.g. MMX or SSE-2);

b) read performance in normal and single failure degraded mode will be comparable to RAID-5;

c) read performance in dual failure degraded mode will be quite bad.

I'm curious how much interest there would be in this, since I certainly have enough projects without it, and I'm probably going to need some of Neil's time to integrate it into the md driver and the tools.

-hpa

Signature · 3 lines
<[email protected]> at work, <[email protected]> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt	<[email protected]>
#1 lore raw
Quoted 4 lines · H. Peter Anvin

I'm curious how much interest there would be in this, since I certainly have enough projects without it, and I'm probably going to need some of Neil's time to integrate it into the md driver and the tools.

There was quite a long thread about this last June ( it starts here
http://marc.theaimsgroup.com/?l=linux-raid&m=102305890732421&w=2). I've seen lack of RAID-6 support cited as one of the shortcoming of Linux SW RAID quite a few times, by quite a few sources. That seems like one reason to implement it.

--
Derek Vadala, [email protected], http://www.cynicism.com/~derek

#2 lore raw
In reply to H. Peter Anvin

Hello Peter ,

Quoted 6 lines · H. Peter Anvin

Hi all,
I'm playing around with RAID-6 algorithms lately. With RAID-6 I mean a setup which needs N+2 disks for N disks worth of storage and can handle any two disks failing -- this seems to be the contemporary definition of RAID-6 (the originally proposed "two-dimensional parity" which required N+2*sqrt(N) drives never took off for obvious reasons.)

Was there a discussion of the 'two-dimensional parity' on the list ? I don't remember any (of course) . But what other than 98+2+10 , What was the main difficulty ? I don't (personally) see any difficulty (other than managability/power/space) to the ammount of disks required . Tia , JimL
--

       +------------------------------------------------------------------+
       | James   W.   Laferriere | System    Techniques | Give me VMS     |
       | Network        Engineer |     P.O. Box 854     |  Give me Linux  |
       | [email protected] | Coudersport PA 16915 |   only  on  AXP |
       +------------------------------------------------------------------+
#3 lore raw
Quoted 11 lines · Mr. James W. Laferriere

Hello Peter ,

Quoted 6 lines · H. Peter Anvin

Hi all,
I'm playing around with RAID-6 algorithms lately. With RAID-6 I mean a setup which needs N+2 disks for N disks worth of storage and can handle any two disks failing -- this seems to be the contemporary definition of RAID-6 (the originally proposed "two-dimensional parity" which required N+2*sqrt(N) drives never took off for obvious reasons.)

Quoted 6 lines

Was there a discussion of the 'two-dimensional parity' on the list ? I don't remember any (of course) . But what other than 98+2+10 , What was the main difficulty ? I don't (personally) see any difficulty (other than managability/power/space) to the ammount of disks required . Tia , JimL

No discussion of two-dimensional parity, but that was the originally proposed RAID-6. Noone ever productized a solution like that to the best of my knowledge. I don't know what you mean with "98+2+10", but the basic problem is that with 2D parity, for N data drives you need 2*sqrt(N) redundancy drives, which for any moderate-sized RAID is a lot (with 9 data drives you need 6 redundancy drives, so you have 67% overhead.) You also have the same kind of performance problems as RAID-4 does, because the "rotating parity" trick of RAID-5 does not work in two dimensions. And for all of this, you're not *guaranteed* more than dual failure recovery (although you might, probabilistically, luck out and have more than that.)

P+Q redundancy, the current meaning of RAID-6, instead uses two orthogonal redundancy functions so you only need two redundancy drives regardless of how much data you have, and you can apply the RAID-5 trick of rotating the parity around. So from your 15 drives in the example above, you get 13 drives worth of data instead of 9.

-hpa

#4 lore raw
In reply to H. Peter Anvin
Quoted 8 lines · H. Peter Anvin

Hi all,

I'm playing around with RAID-6 algorithms lately. With RAID-6 I mean a setup which needs N+2 disks for N disks worth of storage and can handle any two disks failing -- this seems to be the contemporary definition of RAID-6 (the originally proposed "two-dimensional parity" which required N+2*sqrt(N) drives never took off for obvious reasons.)

Quoted 5 lines

Based on my current research, I think the following should be true:

a) write performance will be worse than RAID-5, but I believe it can be kept to within a factor of 1.5-2.0 on machines with suitable SIMD instruction sets (e.g. MMX or SSE-2);

Please note that raw CPU power is usually *not* a limiting (or even significantly contributing) factor, on modern systems.

Limitations are disk reads/writes/seeks, bus bandwidth, etc.

You will probably cause more bus activity with RAID-6, and that might degrade performance. But I don't think you need to worry about MMX/SSE/... If you can do as well as the current RAID-5 code, then you will be in the clear until people have 1GB/sec disk transfer-rates on 500MHz PIII systems ;)

Quoted 3 lines

b) read performance in normal and single failure degraded mode will be comparable to RAID-5;

Which again is like a RAID-0 with some extra seeks... Eg. not too bad with huge chunk sizes.

You might want to consider using huge chunk-sizes when reading, but making sure that writes can be made on "sub-chunks" - so that one could run a RAID-6 with a 128k chunk size, yet have writes performed on 4k chunks. This is important for performance on both read and write, but it is an optimization the current RAID-5 code lacks.

Quoted 7 lines

c) read performance in dual failure degraded mode will be quite bad.

I'm curious how much interest there would be in this, since I certainly have enough projects without it, and I'm probably going to need some of Neil's time to integrate it into the md driver and the tools.

I've seen quite some people ask for it. You might find a friend in "Roy Sigurd Karlsbach" - he for one has been asking (loudly) for it ;)

Go Peter! ;)

Signature · 10 lines
................................................................
:   [email protected]   : And I see the elder races,         :
:.........................: putrid forms of man                :
:   Jakob Østergaard      : See him rise and claim the earth,  :
:        OZ9ABN           : his downfall is at hand.           :
:.........................:............{Konkhra}...............:
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#5 lore raw
Quoted 8 lines · Jakob Oestergaard
Quoted 4 lines

a) write performance will be worse than RAID-5, but I believe it can be kept to within a factor of 1.5-2.0 on machines with suitable SIMD instruction sets (e.g. MMX or SSE-2);

Please note that raw CPU power is usually *not* a limiting (or even significantly contributing) factor, on modern systems.

Quoted 8 lines

Limitations are disk reads/writes/seeks, bus bandwidth, etc.

You will probably cause more bus activity with RAID-6, and that might degrade performance. But I don't think you need to worry about MMX/SSE/... If you can do as well as the current RAID-5 code, then you will be in the clear until people have 1GB/sec disk transfer-rates on 500MHz PIII systems ;)

RAID-6 will, obviously, never do as well as RAID-5 -- you are doing more work (both computational and data-pushing.) The RAID-6 syndrome computation is actually extrememly expensive if you can't do it in parallel. Fortunately there is a way to do it in parallel using MMX or SSE-2, although it seems to exist by pure dumb luck -- certainly not by design. I've tried to figure out how to generalize to using regular 32-bit or 64-bit integer registers, but it doesn't seem to work there.

Again, my initial analysis seems to indicate performance within about a factor of 2.

Quoted 11 lines
Quoted 2 lines

b) read performance in normal and single failure degraded mode will be comparable to RAID-5;

Which again is like a RAID-0 with some extra seeks... Eg. not too bad with huge chunk sizes.

You might want to consider using huge chunk-sizes when reading, but making sure that writes can be made on "sub-chunks" - so that one could run a RAID-6 with a 128k chunk size, yet have writes performed on 4k chunks. This is important for performance on both read and write, but it is an optimization the current RAID-5 code lacks.

That's an issue for the common framework, I'll leave that to Neil. It's functionally equivalent between RAID-5 and -6.

Quoted 9 lines
Quoted 6 lines

c) read performance in dual failure degraded mode will be quite bad.

I'm curious how much interest there would be in this, since I certainly have enough projects without it, and I'm probably going to need some of Neil's time to integrate it into the md driver and the tools.

I've seen quite some people ask for it. You might find a friend in "Roy Sigurd Karlsbach" - he for one has been asking (loudly) for it ;)

:) Enough people have responded that I think I have a project...

-hpa

#6 lore raw
Followup to:  <[email protected]>
By author:    "H. Peter Anvin" <[email protected]>
In newsgroup: linux.dev.raid
Quoted 4 lines

I've tried to figure out how to generalize to using regular 32-bit or 64-bit integer registers, but it doesn't seem to work there.

Of course, once I had said that I just had to go and figure it out :) This is good, because it's inherently the highest performance we can get out of portable code.

-hpa

Signature · 3 lines
<[email protected]> at work, <[email protected]> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt	<[email protected]>
#7 lore raw
In reply to Jakob Oestergaard
Quoted 6 lines · On Tuesday November 12, [email protected] wrote:

You might want to consider using huge chunk-sizes when reading, but making sure that writes can be made on "sub-chunks" - so that one could run a RAID-6 with a 128k chunk size, yet have writes performed on 4k chunks. This is important for performance on both read and write, but it is an optimization the current RAID-5 code lacks.

Either I misunderstand your point, or you misunderstand the code.

A 4k write request will cause a 4k write to a data block and a 4k write to a parity block, no matter what the chunk size is. (There may also be pre-reading, and possibly several 4k writes will share a parity block update).

I see no lacking optimisation, but if you do, I would be keen to hear a more detailed explanation.

NeilBrown

#8 lore raw
Quoted 8 lines · Neil Brown
Quoted 6 lines · On Tuesday November 12, [email protected] wrote:

You might want to consider using huge chunk-sizes when reading, but making sure that writes can be made on "sub-chunks" - so that one could run a RAID-6 with a 128k chunk size, yet have writes performed on 4k chunks. This is important for performance on both read and write, but it is an optimization the current RAID-5 code lacks.

Quoted 6 lines

Either I misunderstand your point, or you misunderstand the code.

A 4k write request will cause a 4k write to a data block and a 4k write to a parity block, no matter what the chunk size is. (There may also be pre-reading, and possibly several 4k writes will share a parity block update).

Writes on a 128k chunk array are significantly slower than writes on a
4k chunk array, according to someone else on this list - I wanted to
look into this myself, but now is just a bad time for me (nothing new on that front).

The benchmark goes:

| some tests on raid5 with 4k and 128k chunk size. The results are as follows:
| Access Spec     4K(MBps)        4K-deg(MBps)    128K(MBps) 128K-deg(MBps)
| 2K Seq Read     23.015089       33.293993       25.415035  32.669278
| 2K Seq Write    27.363041       30.555328       14.185889  16.087862
| 64K Seq Read    22.952559       44.414774       26.02711   44.036993
| 64K Seq Write   25.171833       32.67759        13.97861   15.618126

So down from 27MB/sec to 14MB/sec running 2k-block sequential writes on a 128k chunk array versus a 4k chunk array (non-degraded).

In degraded mode, the writes degenerate from 30MB/sec to 16MB/sec as the chunk-size increases.

Something's fishy.

Quoted 3 lines

I see no lacking optimisation, but if you do, I would be keen to hear a more detailed explanation.

Well if a 4k write really only causes a 4k write to disk, even with a 128k chunk-size array, then something else is happening...

I didn't do the benchmark, and I didn't get to investigate it further here, so I can't really say much else productive :)

/ Jakob "linux-raid message multiplexer" Østergaard

Signature · 10 lines
................................................................
:   [email protected]   : And I see the elder races,         :
:.........................: putrid forms of man                :
:   Jakob Østergaard      : See him rise and claim the earth,  :
:        OZ9ABN           : his downfall is at hand.           :
:.........................:............{Konkhra}...............:
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#9 lore raw
Quoted 15 lines · On Wednesday November 13, [email protected] wrote:

Writes on a 128k chunk array are significantly slower than writes on a
4k chunk array, according to someone else on this list - I wanted to
look into this myself, but now is just a bad time for me (nothing new on that front).

The benchmark goes:

| some tests on raid5 with 4k and 128k chunk size. The results are as follows:
| Access Spec     4K(MBps)        4K-deg(MBps)    128K(MBps) 128K-deg(MBps)
| 2K Seq Read     23.015089       33.293993       25.415035  32.669278
| 2K Seq Write    27.363041       30.555328       14.185889  16.087862
| 64K Seq Read    22.952559       44.414774       26.02711   44.036993
| 64K Seq Write   25.171833       32.67759        13.97861   15.618126
Quoted 2 lines

So down from 27MB/sec to 14MB/sec running 2k-block sequential writes on a 128k chunk array versus a 4k chunk array (non-degraded).

When doing sequential writes, a small chunk size means you are more likely to fill up a whole stripe before data is flushed to disk, so it is very possible that you wont need to pre-read parity at all. With a larger chunksize, it is more likely that you will have to write, and possibly read, the parity block several times.

So if you are doing single threaded sequential accesses, a smaller chunk size is definately better.
If you are doing lots of parallel accesses (typical multi-user work load), small chunk sizes tends to mean that every access goes to all drives so there is lots of contention. In theory a larger chunk size means that more accesses will be entirely satisfied from just one disk, so there it more opportunity for concurrency between the different users.

As always, the best way to choose a chunk size is develop a realistic
work load and test it against several different chunk sizes. There
is no rule like "bigger is better" or "smaller is better".

NeilBrown

#10 lore raw

On Wed, Nov 13, 2002 at 02:33:46PM +1100, Neil Brown wrote:
...

Quoted 9 lines
Quoted 9 lines

The benchmark goes:

| some tests on raid5 with 4k and 128k chunk size. The results are as follows:
| Access Spec     4K(MBps)        4K-deg(MBps)    128K(MBps) 128K-deg(MBps)
| 2K Seq Read     23.015089       33.293993       25.415035  32.669278
| 2K Seq Write    27.363041       30.555328       14.185889  16.087862
| 64K Seq Read    22.952559       44.414774       26.02711   44.036993
| 64K Seq Write   25.171833       32.67759        13.97861   15.618126
Quoted 8 lines
Quoted 2 lines

So down from 27MB/sec to 14MB/sec running 2k-block sequential writes on a 128k chunk array versus a 4k chunk array (non-degraded).

When doing sequential writes, a small chunk size means you are more likely to fill up a whole stripe before data is flushed to disk, so it is very possible that you wont need to pre-read parity at all. With a larger chunksize, it is more likely that you will have to write, and possibly read, the parity block several times.

Except if one worked on 4k sub-chunks - right ? :)

Quoted 3 lines

So if you are doing single threaded sequential accesses, a smaller chunk size is definately better.

Definitely not so for reads - the seeking past the parity blocks ruin sequential read performance when we do many such seeks (eg. when we have small chunks) - as witnessed by the benchmark data above.

Quoted 10 lines

If you are doing lots of parallel accesses (typical multi-user work load), small chunk sizes tends to mean that every access goes to all drives so there is lots of contention. In theory a larger chunk size means that more accesses will be entirely satisfied from just one disk, so there it more opportunity for concurrency between the different users.

As always, the best way to choose a chunk size is develop a realistic
work load and test it against several different chunk sizes. There
is no rule like "bigger is better" or "smaller is better".

For a single reader/writer, it was pretty obvious from the above that "big is good" for reads (because of the fewer parity block skip seeks), and "small is good" for writes.

So, by making a big chunk-sized array, and having it work on 4k sub-chunks for writes, was some idea I had which I felt would just give the best scenario in both cases.

Am I smoking crack, or ? ;)

Signature · 10 lines
................................................................
:   [email protected]   : And I see the elder races,         :
:.........................: putrid forms of man                :
:   Jakob Østergaard      : See him rise and claim the earth,  :
:        OZ9ABN           : his downfall is at hand.           :
:.........................:............{Konkhra}...............:
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#11 lore raw
Quoted 9 lines · Jakob Oestergaard
Quoted 6 lines

When doing sequential writes, a small chunk size means you are more likely to fill up a whole stripe before data is flushed to disk, so it is very possible that you wont need to pre-read parity at all. With a larger chunksize, it is more likely that you will have to write, and possibly read, the parity block several times.

Except if one worked on 4k sub-chunks - right ? :)

No. You probably want to look at the difference between RAID-3 and RAID-4 (RAID-5 being basically RAID-4 twisted around in a rotating pattern.)

Quoted 7 lines

So, by making a big chunk-sized array, and having it work on 4k sub-chunks for writes, was some idea I had which I felt would just give the best scenario in both cases.

Am I smoking crack, or ? ;)

No, you're confusing RAID-3 and RAID-4/5. In RAID-3, sequential blocks are organized as:

	DISKS ------------------------------------>
	 0	 1	 2	 3	PARITY
	 4	 5	 6	 7	PARITY
	 8	 9	10	11	PARITY
	12	13	14	15	PARITY

... whereas in RAID-4 with a chunk size of four blocks it's:

	DISKS ------------------------------------>
	 0	 4	 8	12	PARITY
	 1	 5	 9	13	PARITY
	 2	 6	10	14	PARITY
	 3	 7	11	15	PARITY

If you only write blocks 0-3 you *have* to read in the 12 data blocks and write out all 4 parity blocks, whereas in RAID-3 you can get away with only writing 5 blocks. [Well, technically you could also do a read-modify-write on the parity, since parity is linear. This would greatly complicate the code.]

Therefore, for small sequential writes chunking is *inherently* a lose, and there isn't much you can do about it.

-hpa

#12 lore raw

Peter,

Quoted 9 lines

No, you're confusing RAID-3 and RAID-4/5. In RAID-3, sequential blocks are organized as:

	DISKS ------------------------------------>
	 0	 1	 2	 3	PARITY
	 4	 5	 6	 7	PARITY
	 8	 9	10	11	PARITY
	12	13	14	15	PARITY
Quoted 7 lines

... whereas in RAID-4 with a chunk size of four blocks it's:

	DISKS ------------------------------------>
	 0	 4	 8	12	PARITY
	 1	 5	 9	13	PARITY
	 2	 6	10	14	PARITY
	 3	 7	11	15	PARITY

The description you have for RAID-3 is wrong. What you give as RAID-3 is actually RAID-4 with a 1 block segment size.

RAID-3 uses BITWISE (or BYTEWISE) striping with parity, as opposed to the BLOCKWISE striping with parity in RAID-4/5. In RAID-3, every I/O transaction (regardless of size) accesses all drives in the array (a read doesn't have to access the parity). This gives high transfer rates, even on single-block transactions.

DISKS ------------------------------------>
0.0 0.1 0.2 0.3 PARITY
1.0 1.1 1.2 1.3 PARITY
2.0 2.1 2.2 2.3 PARITY
3.0 3.1 3.2 3.3 PARITY

It is NOT possible to read just 0.0 from the array. If you were to read the raw physical device, the results would be meaningless.

The structure of the array limits the number of spindles to one more than a power of two (3, 5, 9, 17, etc.). I have seen this implemented with 5 and 9 drives (actually, the 9 was done with parallel heads).

Peter Ashford

#13 lore raw
In reply to H. Peter Anvin
Quoted 7 lines · On Wednesday November 13, [email protected] wrote:
	DISKS ------------------------------------>
	 0	 4	 8	12	PARITY
	 1	 5	 9	13	PARITY
	 2	 6	10	14	PARITY
	 3	 7	11	15	PARITY
Quoted 5 lines

If you only write blocks 0-3 you *have* to read in the 12 data blocks and write out all 4 parity blocks, whereas in RAID-3 you can get away with only writing 5 blocks. [Well, technically you could also do a read-modify-write on the parity, since parity is linear. This would greatly complicate the code.]

We do read-modify-write if it involves fewer pre-reads than reconstruct-write.
so in the above scenario, writing blocks 0,1,2,3 would cause a pre-read of those blocks and the 4 parity blocks, and then all 8 blocks would be re-written.

NeilBrown

#14 lore raw
In reply to Jakob Oestergaard

Jakob,

SNIP

Quoted 7 lines

For a single reader/writer, it was pretty obvious from the above that "big is good" for reads (because of the fewer parity block skip seeks), and "small is good" for writes.

So, by making a big chunk-sized array, and having it work on 4k sub-chunks for writes, was some idea I had which I felt would just give the best scenario in both cases.

Actually, the problem is worse than you describe.

Let's assume that we have a RAID-5 array of 5 disks, with a segment size of 64KB. In this instance, the optimum I/O size will be 256KB. Furthermore, that will only be the optimum I/O when it is on a 256KB boundary.

I have, in the past, performed I/O benchmarks on raw arrays (both using the MD driver, and using 3Ware cards). My results show that read speed drops off when the segment size passes 128KB, but write speed stays stable up to 2MB (the largest I/O size I tested).

This information, combined with the benchmarks you posted earlier, shows that the write slowdown when writing large I/O sizes is caused by the file-system structure.

Current Linux file-systems don't support block sizes larger than 4KB. This means that even if you perform the optimum sized I/O, there is no guarantee that the I/O will occur on the optimum boundary (it's actually quite unlikely). To make matters worse, there is no guarantee that when you perform a large write, all the data will be placed in contiguous blocks.

In order to maximize I/O throughput, it will be necessary to create a Linux file-system that can effectively deal with large blocks (not necessarily power of two in size). The alternative would be to work with the raw file-system, as many DBMS' do.

I have worked with a file-system structure that deals well with large blocks, but it is not in the public domain, and I doubt that CRAY is interested in porting the NC1FS structure to Linux.

Peter Ashford

#15 lore raw
In reply to Jakob Oestergaard
Quoted 11 lines · On Wednesday November 13, [email protected] wrote:

On Wed, Nov 13, 2002 at 02:33:46PM +1100, Neil Brown wrote:
...

Quoted 9 lines
Quoted 9 lines

The benchmark goes:

| some tests on raid5 with 4k and 128k chunk size. The results are as follows:
| Access Spec     4K(MBps)        4K-deg(MBps)    128K(MBps) 128K-deg(MBps)
| 2K Seq Read     23.015089       33.293993       25.415035  32.669278
| 2K Seq Write    27.363041       30.555328       14.185889  16.087862
| 64K Seq Read    22.952559       44.414774       26.02711   44.036993
| 64K Seq Write   25.171833       32.67759        13.97861   15.618126

These numbers look ... interesting. I might try to reproduce them myself.

Quoted 10 lines
Quoted 8 lines
Quoted 2 lines

So down from 27MB/sec to 14MB/sec running 2k-block sequential writes on a 128k chunk array versus a 4k chunk array (non-degraded).

When doing sequential writes, a small chunk size means you are more likely to fill up a whole stripe before data is flushed to disk, so it is very possible that you wont need to pre-read parity at all. With a larger chunksize, it is more likely that you will have to write, and possibly read, the parity block several times.

Except if one worked on 4k sub-chunks - right ? :)

I still don't understand.... We *do* work with 4k subchunks.

Quoted 8 lines
Quoted 4 lines

So if you are doing single threaded sequential accesses, a smaller chunk size is definately better.

Definitely not so for reads - the seeking past the parity blocks ruin sequential read performance when we do many such seeks (eg. when we have small chunks) - as witnessed by the benchmark data above.

Parity blocks aren't big enough to have to seek past. I would imagine that a modern drive would read a whole track into cache on the first read request, and then find the required data, just past the parity block, in the cache on the second request. By maybe I'm wrong.

Or there could be some factor in the device driver where lots of little read request, even though they are almost consecutive, are handled more poorly than a few large read requests.
I wonder if it would be worth reading those parity blocks anyway if a sequential read were detected....

Quoted 12 lines
Quoted 11 lines

If you are doing lots of parallel accesses (typical multi-user work load), small chunk sizes tends to mean that every access goes to all drives so there is lots of contention. In theory a larger chunk size means that more accesses will be entirely satisfied from just one disk, so there it more opportunity for concurrency between the different users.

As always, the best way to choose a chunk size is develop a realistic
work load and test it against several different chunk sizes. There
is no rule like "bigger is better" or "smaller is better".

Quoted 7 lines

For a single reader/writer, it was pretty obvious from the above that "big is good" for reads (because of the fewer parity block skip seeks), and "small is good" for writes.

So, by making a big chunk-sized array, and having it work on 4k sub-chunks for writes, was some idea I had which I felt would just give the best scenario in both cases.

The issue isn't so much the IO size as the layout of disk. You cannot use one layout for read and a different layout for write. That obviously doesn't make sense.

NeilBrown