MD: improve raid1/10 write performance for fast storage

In raid1/10, all write requests are dispatched in a single thread. In fast storage, the thread is a bottleneck, because it dispatches request too slow. Also the thread migrates freely, which makes request completion cpu not match with submission cpu even driver/block layer has such capability. This will cause bad cache issue. Both these are not a big deal for slow storage.

Switching the dispatching to percpu/perthread based dramatically increases performance. The more raid disk number is, the more performance boosts. In a 4-disk raid10 setup, this can double the throughput.

percpu/perthread based dispatch doesn't harm slow storage. This is the way how raw device is accessed, and there is correct block plug set which can help do request merge and reduce lock contention.

V2->V3:
rebase to latest tree and fix cpuhotplug issue

V1->V2:
1. droped direct dispatch patches. That has better performance imporvement, but is hopelessly made correct.
2. Add a MD specific workqueue to do percpu dispatch.

Quoted 17 lines · On Wed, 13 Jun 2012 17:11:43 +0800 Shaohua Li <[email protected]> wrote:

In raid1/10, all write requests are dispatched in a single thread. In fast storage, the thread is a bottleneck, because it dispatches request too slow. Also the thread migrates freely, which makes request completion cpu not match with submission cpu even driver/block layer has such capability. This will cause bad cache issue. Both these are not a big deal for slow storage.

Switching the dispatching to percpu/perthread based dramatically increases performance. The more raid disk number is, the more performance boosts. In a 4-disk raid10 setup, this can double the throughput.

percpu/perthread based dispatch doesn't harm slow storage. This is the way how raw device is accessed, and there is correct block plug set which can help do request merge and reduce lock contention.

V2->V3:
rebase to latest tree and fix cpuhotplug issue

Quoted 4 lines

V1->V2:
1. droped direct dispatch patches. That has better performance imporvement, but is hopelessly made correct.
2. Add a MD specific workqueue to do percpu dispatch.

I still don't like the per-cpu allocations and the extra work queues.

Quoted 23 lines · NeilBrown

Quoted 21 lines · On Wed, 13 Jun 2012 17:11:43 +0800 Shaohua Li <[email protected]> wrote:

In raid1/10, all write requests are dispatched in a single thread. In fast storage, the thread is a bottleneck, because it dispatches request too slow. Also the thread migrates freely, which makes request completion cpu not match with submission cpu even driver/block layer has such capability. This will cause bad cache issue. Both these are not a big deal for slow storage.

Switching the dispatching to percpu/perthread based dramatically increases performance. The more raid disk number is, the more performance boosts. In a 4-disk raid10 setup, this can double the throughput.

percpu/perthread based dispatch doesn't harm slow storage. This is the way how raw device is accessed, and there is correct block plug set which can help do request merge and reduce lock contention.

V2->V3:
rebase to latest tree and fix cpuhotplug issue

V1->V2:
1. droped direct dispatch patches. That has better performance imporvement, but is hopelessly made correct.
2. Add a MD specific workqueue to do percpu dispatch.

Quoted 1 line

I still don't like the per-cpu allocations and the extra work queues.

Why don't you like this method Neil? Complexity? The performance seems to be there.

Quoted 26 lines · On Thu, 28 Jun 2012 20:29:21 -0500 Stan Hoeppner <[email protected]> wrote:

Quoted 23 lines · NeilBrown

Quoted 21 lines · On Wed, 13 Jun 2012 17:11:43 +0800 Shaohua Li <[email protected]> wrote:

In raid1/10, all write requests are dispatched in a single thread. In fast storage, the thread is a bottleneck, because it dispatches request too slow. Also the thread migrates freely, which makes request completion cpu not match with submission cpu even driver/block layer has such capability. This will cause bad cache issue. Both these are not a big deal for slow storage.

Switching the dispatching to percpu/perthread based dramatically increases performance.  The more raid disk number is, the more performance boosts. In a 4-disk raid10 setup, this can double the throughput.

percpu/perthread based dispatch doesn't harm slow storage. This is the way how raw device is accessed, and there is correct block plug set which can help do request merge and reduce lock contention.

V2->V3:
rebase to latest tree and fix cpuhotplug issue

V1->V2:
1. droped direct dispatch patches. That has better performance imporvement, but is hopelessly made correct.
2. Add a MD specific workqueue to do percpu dispatch.

Quoted 5 lines

Quoted 1 line

I still don't like the per-cpu allocations and the extra work queues.

Why don't you like this method Neil?  Complexity?  The performance seems to be there.

Not an easy question to answer.  It just doesn't "taste" nice.
I certainly like the performance and if this is the only way to get that performance then we'll probably go that way.  But I'm not convinced it is the only way and I want to explore other options first.
I guess it feels a bit heavy handed.  On machines with 1024 cores, per-cpu allocations and per-cpu threads are not as cheap as they are one 2-core machines.  And I'm hoping for a 1024-core phone soon :-)

NeilBrown

Quoted 28 lines · On Thu, 28 Jun 2012 20:29:21 -0500 Stan Hoeppner <[email protected]> wrote:

Quoted 26 lines · NeilBrown

Quoted 14 lines · On Wed, 13 Jun 2012 17:11:43 +0800 Shaohua Li <[email protected]> wrote:

In raid1/10, all write requests are dispatched in a single thread. In fast storage, the thread is a bottleneck, because it dispatches request too slow. Also the thread migrates freely, which makes request completion cpu not match with submission cpu even driver/block layer has such capability. This will cause bad cache issue. Both these are not a big deal for slow storage.

Switching the dispatching to percpu/perthread based dramatically increases performance. The more raid disk number is, the more performance boosts. In a 4-disk raid10 setup, this can double the throughput.

percpu/perthread based dispatch doesn't harm slow storage. This is the way how raw device is accessed, and there is correct block plug set which can help do request merge and reduce lock contention.

Quoted 7 lines

V2->V3:
rebase to latest tree and fix cpuhotplug issue

V1->V2:
1. droped direct dispatch patches. That has better performance imporvement, but is hopelessly made correct.
2. Add a MD specific workqueue to do percpu dispatch.

I still don't like the per-cpu allocations and the extra work queues.

Quoted 3 lines

Why don't you like this method Neil? Complexity? The performance seems to be there.

Not an easy question to answer. It just doesn't "taste" nice.
I certainly like the performance and if this is the only way to get that performance then we'll probably go that way. But I'm not convinced it is the only way and I want to explore other options first.

I guess it feels a bit heavy handed. On machines with 1024 cores, per-cpu allocations and per-cpu threads are not as cheap as they are one 2-core machines. And I'm hoping for a 1024-core phone soon :-)

Quoted 15 lines

In raid1/10, all write requests are dispatched in a single thread. In fast storage, the thread is a bottleneck, because it dispatches request too slow. Also the thread migrates freely, which makes request completion cpu not match with submission cpu even driver/block layer has such capability. This will cause bad cache issue. Both these are not a big deal for slow storage.

Switching the dispatching to percpu/perthread based dramatically increases performance.  The more raid disk number is, the more performance boosts. In a 4-disk raid10 setup, this can double the throughput.

percpu/perthread based dispatch doesn't harm slow storage. This is the way how raw device is accessed, and there is correct block plug set which can help do request merge and reduce lock contention.

Quoted 7 lines

V2->V3:
rebase to latest tree and fix cpuhotplug issue

V1->V2:
1. droped direct dispatch patches. That has better performance imporvement, but is hopelessly made correct.
2. Add a MD specific workqueue to do percpu dispatch.

Hi.

I still don't like the per-cpu allocations and the extra work queues.

The following patch demonstrates how I would like to address this issue.  It should submit requests from the same thread that initially made the request - at least in most cases.

It leverages the plugging code and pushed everything out on the unplug, unless that comes from a scheduler call (which should be uncommon).  In that case it falls back on passing all the requests to the md thread.

Obviously if we proceed with this I'll split this up into neat reviewable patches.  However before that it would help to know if it really helps as I think it should.

So would you be able to test it on your SSD hardware and see how it compares the current code, and to you code?  Thanks.

I have only tested it lightly myself so there could still be bugs, but hopefully not obvious ones.

A simple "time mkfs" test on very modest hardware show as 25% reduction in total time (168s -> 127s).  I guess that's a 33% increase in speed? However sequential writes with 'dd' seem a little slower (14MB/s -> 13.6MB/s)

There are some hacks in there that need to be cleaned up, but I think the general structure looks good.

2012/6/28 NeilBrown <[email protected]>:

Quoted 51 lines · On Wed, 13 Jun 2012 17:11:43 +0800 Shaohua Li <[email protected]> wrote:

Quoted 15 lines

In raid1/10, all write requests are dispatched in a single thread. In fast storage, the thread is a bottleneck, because it dispatches request too slow. Also the thread migrates freely, which makes request completion cpu not match with submission cpu even driver/block layer has such capability. This will cause bad cache issue. Both these are not a big deal for slow storage.

Switching the dispatching to percpu/perthread based dramatically increases performance.  The more raid disk number is, the more performance boosts. In a 4-disk raid10 setup, this can double the throughput.

percpu/perthread based dispatch doesn't harm slow storage. This is the way how raw device is accessed, and there is correct block plug set which can help do request merge and reduce lock contention.

Quoted 7 lines

V2->V3:
rebase to latest tree and fix cpuhotplug issue

V1->V2:
1. droped direct dispatch patches. That has better performance imporvement, but is hopelessly made correct.
2. Add a MD specific workqueue to do percpu dispatch.

Hi.

I still don't like the per-cpu allocations and the extra work queues.

The following patch demonstrates how I would like to address this issue.  It should submit requests from the same thread that initially made the request - at least in most cases.

It leverages the plugging code and pushed everything out on the unplug, unless that comes from a scheduler call (which should be uncommon).  In that case it falls back on passing all the requests to the md thread.

Obviously if we proceed with this I'll split this up into neat reviewable patches.  However before that it would help to know if it really helps as I think it should.

So would you be able to test it on your SSD hardware and see how it compares the current code, and to you code?  Thanks.

I have only tested it lightly myself so there could still be bugs, but hopefully not obvious ones.

A simple "time mkfs" test on very modest hardware show as 25% reduction in total time (168s -> 127s).  I guess that's a 33% increase in speed? However sequential writes with 'dd' seem a little slower (14MB/s -> 13.6MB/s)

Quoted 2 lines

There are some hacks in there that need to be cleaned up, but I think the general structure looks good.

Thought I consider this approach before, and schedule from the unplug callback is an issue. Maybe I overlooked it at that time, the from_schedule check looks promising.

Quoted 52 lines · Shaohua Li

2012/6/28 NeilBrown <[email protected]>:

Quoted 51 lines · On Wed, 13 Jun 2012 17:11:43 +0800 Shaohua Li <[email protected]> wrote:

Quoted 15 lines

In raid1/10, all write requests are dispatched in a single thread. In fast storage, the thread is a bottleneck, because it dispatches request too slow. Also the thread migrates freely, which makes request completion cpu not match with submission cpu even driver/block layer has such capability. This will cause bad cache issue. Both these are not a big deal for slow storage.

Switching the dispatching to percpu/perthread based dramatically increases performance.  The more raid disk number is, the more performance boosts. In a 4-disk raid10 setup, this can double the throughput.

percpu/perthread based dispatch doesn't harm slow storage. This is the way how raw device is accessed, and there is correct block plug set which can help do request merge and reduce lock contention.

Quoted 7 lines

V2->V3:
rebase to latest tree and fix cpuhotplug issue

V1->V2:
1. droped direct dispatch patches. That has better performance imporvement, but is hopelessly made correct.
2. Add a MD specific workqueue to do percpu dispatch.

Hi.

I still don't like the per-cpu allocations and the extra work queues.

The following patch demonstrates how I would like to address this issue.  It should submit requests from the same thread that initially made the request - at least in most cases.

It leverages the plugging code and pushed everything out on the unplug, unless that comes from a scheduler call (which should be uncommon).  In that case it falls back on passing all the requests to the md thread.

Obviously if we proceed with this I'll split this up into neat reviewable patches.  However before that it would help to know if it really helps as I think it should.

So would you be able to test it on your SSD hardware and see how it compares the current code, and to you code?  Thanks.

I have only tested it lightly myself so there could still be bugs, but hopefully not obvious ones.

A simple "time mkfs" test on very modest hardware show as 25% reduction in total time (168s -> 127s).  I guess that's a 33% increase in speed? However sequential writes with 'dd' seem a little slower (14MB/s -> 13.6MB/s)

Quoted 6 lines

Quoted 2 lines

There are some hacks in there that need to be cleaned up, but I think the general structure looks good.

Thought I consider this approach before, and schedule from the unplug callback is an issue. Maybe I overlooked it at that time, the from_schedule check looks promising.

I tried raid1/raid10 performance with this patch (with similar change for raid10, and add plug in the raid1/10 unplug function for dispatching), the result is ok. The from_schedule check does the trick, there isn't race I mentioned before. And I double checked the rate unplug is called from schedule, which is very very low.

Now the only problem is if extra bitmap flush could be an overhead. Our card hasn't such overhead, so not sure.

Looks you merged the patch to your tree, great! The raid1_unplug() still lacks blk_start_plug/blk_finish_plug.

Will you add a similar patch for raid10?