IO scheduler, queue depth, nr_requests

Hello,

as you might have seen from the linux-kernel mailinglist
I have been testing for months now with a fileserver set up to use XFS over LVM2 on a 3ware RAID5 controller.

I asked for help several times on the list, but nobody really replied, so now I'm taking a shot at mailing you directly, since you appear to be the I/O request queueing guru of the kernel ;)
Cc: sent to [email protected]. Any hint appreciated.

For some reason, when using LVM, write requests get queued out of order to the 3ware controller, which results in quite a bit of seeking and thus performance loss.

The default queue depth of the 3ware controller is 254. I found out that lowering it to 64 in the driver fixed my problems, and I advised 3ware support about this. They weren't really convinced..

By fiddling about today I just found that changing /sys/block/sda/queue/nr_requests from 128 to something above the queue depth of the 3ware controller (256 doesn't work, 384 and up do) also fixes the problem.

Does that actually make sense ?

Ah yes, I'm currently using 2.6.2 with a 3ware 8506-8 in hardware raid5 mode, deadline scheduler, PIV 3.0 Ghz, 2 GB RAM.

Debug output ("mydd" works just like "dd", but has an fsync option):

- /mnt is an XFS filesystem on a LVM2 volume on the 3ware
- /mnt2 is an XFS filesystem directly on /dev/sda1 of the 3ware

- First on /mnt, the LVM partition. Note that a small "dd" runs fast, a larger one runs slower:

# cd /mnt
# cat /sys/block/sda/device/queue_depth
254
# cat /sys/block/sda/queue/nr_requests
128
# ~/mydd --if /dev/zero --of file --bs 4096 --count 50000 --fsync
204800000 bytes transferred in 2.679812 seconds (76423271 bytes/sec)
# ~/mydd --if /dev/zero --of file --bs 4096 --count 100000 --fsync
409600000 bytes transferred in 9.501549 seconds (43108760 bytes/sec)

- Now I set the nr_requests to 512:

# echo 512 > /sys/block/sda/queue/nr_requests
# ~/mydd --if /dev/zero --of file --bs 4096 --count 100000 --fsync
409600000 bytes transferred in 5.374437 seconds (76212634 bytes/sec)

See that ? Weird thing is, it's only on LVM, directly on /dev/sda1 no problem at all:

# cat /sys/block/sda/device/queue_depth
254
# cat /sys/block/sda/queue/nr_requests
128
# ~/mydd --if /dev/zero --of file --bs 4096 --count 100000 --fsync
409600000 bytes transferred in 5.135642 seconds (79756338 bytes/sec)

Somehow, LVM is causing the requests to the underlying 3ware device to get out of order, and increasing nr_requests to be larger than the queue_depth of the device fixes this.

I tried the latest dm-patches in -mm (applied those to vanilla 2.6.2), which include a patch called dm-04-maintain-bio-ordering.patch but that doesn't really help (at first I though otherwise, but the tests scripts I used lowered the queue_depth of the 3ware to 64 by accident) - if anything, it makes things worse.

# ~/mydd --if /dev/zero --of file --bs 4096 --count 100000 --fsync 409600000 bytes transferred in 13.138224 seconds (31176208 bytes/sec)

Setting nr_requests to 512 fixes things up again.

Seems there's an extra problem here, the nr_requests vs depth problem should not be too problematic unless you have heavy random io. Doesn't look like dm is reordering (bio_list_add() adds to tail, flush_deferred_io() processes from head. direct queueing doesn't look like it's reordering). Can the dm folks verify this?

Quoted 6 lines · Miquel van Smoorenburg

Hello,

as you might have seen from the linux-kernel mailinglist
I have been testing for months now with a fileserver set up to use XFS over LVM2 on a 3ware RAID5 controller.

Quoted 4 lines

By fiddling about today I just found that changing /sys/block/sda/queue/nr_requests from 128 to something above the queue depth of the 3ware controller (256 doesn't work, 384 and up do) also fixes the problem.

Quoted 4 lines

By fiddling about today I just found that changing /sys/block/sda/queue/nr_requests from 128 to something above the queue depth of the 3ware controller (256 doesn't work, 384 and up do) also fixes the problem.

Quoted 5 lines · Miquel van Smoorenburg

By fiddling about today I just found that changing /sys/block/sda/queue/nr_requests from 128 to something above the queue depth of the 3ware controller (256 doesn't work, 384 and up do) also fixes the problem.

Quoted 10 lines

See that ? Weird thing is, it's only on LVM, directly on /dev/sda1 no problem at all:

# cat /sys/block/sda/device/queue_depth
254
# cat /sys/block/sda/queue/nr_requests
128
# ~/mydd --if /dev/zero --of file --bs 4096 --count 100000 --fsync
409600000 bytes transferred in 5.135642 seconds (79756338 bytes/sec)

Quoted 3 lines

Somehow, LVM is causing the requests to the underlying 3ware device to get out of order, and increasing nr_requests to be larger than the queue_depth of the device fixes this.

Seems there's an extra problem here, the nr_requests vs depth problem should not be too problematic unless you have heavy random io. Doesn't look like dm is reordering (bio_list_add() adds to tail, flush_deferred_io() processes from head. direct queueing doesn't look like it's reordering). Can the dm folks verify this?

Or, you are just being hit by the problem first listed - requests get no hold time in the io scheduler for merging, because the driver drains them too quickly because of this artificially huge queue depth. If you did some stats on average request size and io/sec rate that should tell you for sure. I don't know what you have behind the 3ware, but it's generally not advised to use more than 4 tags per spindle.

For some reason, when using LVM, write requests get queued out of order to the 3ware controller, which results in quite a bit of seeking and thus performance loss.

Okay I repeated some earlier tests, and I added some debug code in several places.

I added logging to tw_scsi_queue() in the 3ware driver to log the start sector and length of each request. It logs something like:
3wdbg: id 119, lba = 0x2330bc33, num_sectors = 256

With a perl script, I can check if the requests are sent to the host in order. That outputs something like this:

Consecutive: start 1180906348, length 7936 sec (3968 KB), requests: 31
Consecutive: start 1180906340, length 8 sec (4 KB), requests: 1
Consecutive: start 1180914292, length 7936 sec (3968 KB), requests: 31
Consecutive: start 1180914284, length 8 sec (4 KB), requests: 1
Consecutive: start 1180922236, length 7936 sec (3968 KB), requests: 31
Consecutive: start 1180922228, length 8 sec (4 KB), requests: 1
Consecutive: start 1180930180, length 7936 sec (3968 KB), requests: 31

See, 31 requests in order, then one request "backwards", then 31 in order, etc.

Quoted 3 lines · Miquel van Smoorenburg

For some reason, when using LVM, write requests get queued out of order to the 3ware controller, which results in quite a bit of seeking and thus performance loss.

[..]

Quoted 10 lines

Okay I repeated some earlier tests, and I added some debug code in several places.

I added logging to tw_scsi_queue() in the 3ware driver to log the start sector and length of each request. It logs something like:
3wdbg: id 119, lba = 0x2330bc33, num_sectors = 256

With a perl script, I can check if the requests are sent to the host in order. That outputs something like this:

Quoted 7 lines

Consecutive: start 1180906348, length 7936 sec (3968 KB), requests: 31
Consecutive: start 1180906340, length 8 sec (4 KB), requests: 1
Consecutive: start 1180914292, length 7936 sec (3968 KB), requests: 31
Consecutive: start 1180914284, length 8 sec (4 KB), requests: 1
Consecutive: start 1180922236, length 7936 sec (3968 KB), requests: 31
Consecutive: start 1180922228, length 8 sec (4 KB), requests: 1
Consecutive: start 1180930180, length 7936 sec (3968 KB), requests: 31

Quoted 2 lines

See, 31 requests in order, then one request "backwards", then 31 in order, etc.

I found out what causes this. It's get_request_wait().

When the request queue is full, and a new request needs to be created, __make_request() blocks in get_request_wait().

Another process wakes up first (pdflush / process submitting I/O itself / xfsdatad / etc) and sends the next bio's to __make_request(). In the mean time some free requests have become available, and the bios are merged into a new request. Those requests are submitted to the device.

Then, get_request_wait() returns but the bio is not mergeable anymore - and that results in a backwards seek, severely limiting the I/O rate.

Wouldn't it be better to allow the request allocation and queue the request, and /then/ put the process to sleep ? The queue will grow larger than nr_requests, but it does that anyway.

Quoted 16 lines · Miquel van Smoorenburg

Quoted 3 lines · Miquel van Smoorenburg

For some reason, when using LVM, write requests get queued out of order to the 3ware controller, which results in quite a bit of seeking and thus performance loss.

[..]

Quoted 10 lines

Okay I repeated some earlier tests, and I added some debug code in several places.

I added logging to tw_scsi_queue() in the 3ware driver to log the start sector and length of each request. It logs something like:
3wdbg: id 119, lba = 0x2330bc33, num_sectors = 256

With a perl script, I can check if the requests are sent to the host in order. That outputs something like this:

Quoted 7 lines

Quoted 7 lines

Consecutive: start 1180906348, length 7936 sec (3968 KB), requests: 31
Consecutive: start 1180906340, length 8 sec (4 KB), requests: 1
Consecutive: start 1180914292, length 7936 sec (3968 KB), requests: 31
Consecutive: start 1180914284, length 8 sec (4 KB), requests: 1
Consecutive: start 1180922236, length 7936 sec (3968 KB), requests: 31
Consecutive: start 1180922228, length 8 sec (4 KB), requests: 1
Consecutive: start 1180930180, length 7936 sec (3968 KB), requests: 31

3968 / 31 = 128 exactly. I think when you say "requests: 31" you are actually referring to a single request which has 31 128k BIOs in it, yes?

Quoted 12 lines

Quoted 1 line

See, 31 requests in order, then one request "backwards", then 31 in order, etc.

I found out what causes this. It's get_request_wait().

When the request queue is full, and a new request needs to be created, __make_request() blocks in get_request_wait().

Another process wakes up first (pdflush / process submitting I/O itself / xfsdatad / etc) and sends the next bio's to __make_request(). In the mean time some free requests have become available, and the bios are merged into a new request. Those requests are submitted to the device.

Quoted 6 lines

Then, get_request_wait() returns but the bio is not mergeable anymore - and that results in a backwards seek, severely limiting the I/O rate.

Wouldn't it be better to allow the request allocation and queue the request, and /then/ put the process to sleep ? The queue will grow larger than nr_requests, but it does that anyway.

That would help, but is a bit kludgy.

What _should_ have happened was:

a) The queue gets plugged

b) The maximal-sized 31-bio request is queued

c) The 4k request gets inserted *before* the 3968k request

d) The queue gets unplugged, and both requests stream smoothly.

(And this assumes that the queue was initially empty. ALmost certainly that was not the case).

The thing is, the bio's are submitted perfectly sequentially. It's just that
every so often a request (with just its initial bio) gets stuck for a while. Because the bio's after it are merged and sent to the device, it's not possible to merge that stuck request later on when it gets unstuck, because the other bio's have already left the building so to speak.

Quoted 6 lines · Miquel van Smoorenburg

The thing is, the bio's are submitted perfectly sequentially. It's just that
every so often a request (with just its initial bio) gets stuck for a while. Because the bio's after it are merged and sent to the device, it's not possible to merge that stuck request later on when it gets unstuck, because the other bio's have already left the building so to speak.

Oh. So the raid controller's queue depth is larger than the kernel's. So everything gets immediately shovelled into the device and the kernel is left with nothing to merge the little request against.

Shouldn't the controller itself be performing the insertion?

Quoted 8 lines · Andrew Morton

Quoted 6 lines · Miquel van Smoorenburg

The thing is, the bio's are submitted perfectly sequentially. It's just that
every so often a request (with just its initial bio) gets stuck for a while. Because the bio's after it are merged and sent to the device, it's not possible to merge that stuck request later on when it gets unstuck, because the other bio's have already left the building so to speak.

Quoted 3 lines

Oh. So the raid controller's queue depth is larger than the kernel's. So everything gets immediately shovelled into the device and the kernel is left with nothing to merge the little request against.

Well, the request queue of the kernel is max'ed out too, otherwise get_request_wait() wouldn't be called. It's just an unfortunate timing issue.

Quoted 1 line

Shouldn't the controller itself be performing the insertion?

Well, you would indeed expect the 3ware hardware to be smarter than that, but in its defence, the driver doesn't set sdev->simple_tags or sdev->ordered_tags at all. It just has a large queue on the host, in hardware.

Perhaps this info should be exported into the request queue of the device, so that ll_rw_blk knows about this and can do something similar to the hack I posted ?

Note that AFAICS nothing in drivers/scsi uses the tagging stuff in ll_rw_blk.c. blk_queue_init_tags() is only called by scsi_activate_tcq(), and nothing ever calls that (except the 53c700.c driver). So you can't just check for QUEUE_FLAG_QUEUED. Hmm, nothing in drivers/block calls it either. It's not being used at all yet ? Or am I being dense ?

Quoted 2 lines · Miquel van Smoorenburg

Quoted 1 line

Shouldn't the controller itself be performing the insertion?

Quoted 4 lines

Well, you would indeed expect the 3ware hardware to be smarter than that, but in its defence, the driver doesn't set sdev->simple_tags or sdev->ordered_tags at all. It just has a large queue on the host, in hardware.

A too large queue. IMHO the simple and correct solution to your problem is to diminish the host queue (sane solution), or bump the block layer queue size (dumb solution).

Quoted 5 lines · Jens Axboe

Quoted 2 lines · Miquel van Smoorenburg

Quoted 2 lines

Shouldn't the controller itself be performing the insertion?

Quoted 9 lines

Quoted 5 lines

Well, you would indeed expect the 3ware hardware to be smarter than that, but in its defence, the driver doesn't set sdev->simple_tags or sdev->ordered_tags at all. It just has a large queue on the host, in hardware.

A too large queue. IMHO the simple and correct solution to your problem is to diminish the host queue (sane solution), or bump the block layer queue size (dumb solution).

Well, I did that. Lowering the queue size of the 3ware controller to 64 does help a bit, but performance is still not optimal - leaving it at 254 and increasing the nr_requests of the queue to 512 helps the most.

But the patch I posted does just as well, without any tuning. I changed it a little though - it only has the "new" behaviour (instead of blocking on allocating a request, allocate it, queue it, _then_ block) for WRITEs. That results in the best performance I've seen, by far.

Now the style of my patch might be ugly, but what is conceptually wrong with allocating the request and queueing it, then block if the queue is full, versus blocking on allocating the request and keeping a bio "stuck" for quite some time, resulting in out-of-order requests to the hardware ?

Note that this is not an issue of '2 processes writing to 1 file', really. It's one process and pdflush writing the same dirty pages of the same file.

Quoted 17 lines · Miquel van Smoorenburg

Quoted 5 lines · Jens Axboe

Quoted 2 lines · Miquel van Smoorenburg

Quoted 2 lines

Shouldn't the controller itself be performing the insertion?

Quoted 9 lines

Quoted 5 lines

Well, you would indeed expect the 3ware hardware to be smarter than that, but in its defence, the driver doesn't set sdev->simple_tags or sdev->ordered_tags at all. It just has a large queue on the host, in hardware.

A too large queue. IMHO the simple and correct solution to your problem is to diminish the host queue (sane solution), or bump the block layer queue size (dumb solution).

Quoted 10 lines

Well, I did that. Lowering the queue size of the 3ware controller to 64 does help a bit, but performance is still not optimal - leaving it at 254 and increasing the nr_requests of the queue to 512 helps the most.

But the patch I posted does just as well, without any tuning. I changed it a little though - it only has the "new" behaviour (instead of blocking on allocating a request, allocate it, queue it, _then_ block) for WRITEs. That results in the best performance I've seen, by far.

That's because you are half introducing per-process limits.

Quoted 7 lines

Now the style of my patch might be ugly, but what is conceptually wrong with allocating the request and queueing it, then block if the queue is full, versus blocking on allocating the request and keeping a bio "stuck" for quite some time, resulting in out-of-order requests to the hardware ?

Conceptually? The concept that you have everything you need to continue and yet you block anyway is wrong.

Quoted 2 lines

Note that this is not an issue of '2 processes writing to 1 file', really. It's one process and pdflush writing the same dirty pages of the same file.

pdflush is a process though, that's all that matters.

Quoted 3 lines · Nick Piggin

Miquel van Smoorenburg wrote:

Quoted 5 lines

Quoted 3 lines

Note that this is not an issue of '2 processes writing to 1 file', really. It's one process and pdflush writing the same dirty pages of the same file.

pdflush is a process though, that's all that matters.

I understand that when the two processes are unrelated, the patch as I sent it will do the wrong thing.

But the thing is, you get this:

- "dd" process writes requests
- pdflush triggers to write dirty pages
- too many pages are dirty so "dd" blocks as well to write synchronously
- "dd" process triggers "queue full" but gets marked as "batching" so can continue (get_request)
- pdflush tries to submit one bio and gets blocked (get_request_wait)
- "dd" continues, but that one bio from pdflush remains stuck for a while

That's stupid, that one bio from pdflush should really be allowed on the queue, since "dd" is adding requests from the same source to it anyway.

Perhaps writes from pdflush should be handled differently to prevent this specific case ?

Say, if pdflush adds request #128, don't mark it as batching, but let it block. The next process will be the one marked as batching and can continue. If pdflush tries to add a request > 128, allow it, but _then_ block it.

Would something like that work ? Would it be a good idea to never mark a pdflush process as batching, or would that have a negative impact for some things ?

Even with this patch, it might still be a good idea to allow pdflush to disregard the limits...

Quoted 3 lines · Nick Piggin

Even with this patch, it might still be a good idea to allow pdflush to disregard the limits...

Has it been confirmed that pdflush is blocking in get_request_wait()? I guess that can happen very occasionally because we don't bother with any locking around there but if it's happening a lot then something is bust.

Quoted 11 lines · Andrew Morton

Quoted 3 lines · Nick Piggin

Even with this patch, it might still be a good idea to allow pdflush to disregard the limits...

Has it been confirmed that pdflush is blocking in get_request_wait()? I guess that can happen very occasionally because we don't bother with any locking around there but if it's happening a lot then something is bust.

Miquel's analysis is pretty plausible, but I'm not sure if he's confirmed it or not, Miquel? Even if it isn't happening a lot, and something isn't bust it might be a good idea to do this.

Quoted 11 lines · Nick Piggin

Quoted 6 lines · Andrew Morton

Quoted 3 lines · Nick Piggin

Even with this patch, it might still be a good idea to allow pdflush to disregard the limits...

Quoted 8 lines

Quoted 5 lines

Has it been confirmed that pdflush is blocking in get_request_wait()? I guess that can happen very occasionally because we don't bother with any locking around there but if it's happening a lot then something is bust.

Miquel's analysis is pretty plausible, but I'm not sure if he's confirmed it or not, Miquel?

Quoted 3 lines

Even if it isn't happening
a lot, and something isn't bust it might be a good idea to do this.

Seems OK from a quick check. pdflush will block in get_request_wait() occasionally, but not at all often. Perhaps we could move the write_congested test into the mpage_writepages() inner loop but it hardly seems worth the risk.

Maybe things are different on Miquel's clockwork controller.

 drivers/block/ll_rw_blk.c |    2 ++
 fs/fs-writeback.c         |    2 ++
 2 files changed, 4 insertions(+)

drivers/md/dm-table.c renamed from drivers/md/dm-table.c.ORIG +0 -0

drivers/md/dm-table.c line 878

Quoted 5 lines

drivers/md/dm-table.c renamed from drivers/md/dm-table.c.ORIG +5 -0

+	devices = dm_table_get_devices(t);
+	for (d = devices->next; d != devices; d = d->next) {
+		struct dm_dev *dd = list_entry(d, struct dm_dev, list);
+		request_queue_t *q = bdev_get_queue(dd->bdev);
+		r |= test_bit(bdi_state, &(q->backing_dev_info.state));

Shouldn't this be calling your bdi_*_congested function rather than assuming it is a real device under dm ? (often not true).

I'm also very slightly worried that or'ing together the congestion results for all the seperate devices isn't always the right thing. These devices include anything that the targets are using, exception stores for snapshots, logs for mirror, all paths for multipath (or'ing is most likely to be wrong for multipath).

Quoted 9 lines · Joe Thornber

drivers/md/dm-table.c line 878

Quoted 5 lines

drivers/md/dm-table.c renamed from drivers/md/dm-table.c.ORIG +5 -0

+	devices = dm_table_get_devices(t);
+	for (d = devices->next; d != devices; d = d->next) {
+		struct dm_dev *dd = list_entry(d, struct dm_dev, list);
+		request_queue_t *q = bdev_get_queue(dd->bdev);
+		r |= test_bit(bdi_state, &(q->backing_dev_info.state));

Shouldn't this be calling your bdi_*_congested function rather than assuming it is a real device under dm ? (often not true).

Quoted 5 lines

I'm also very slightly worried that or'ing together the congestion results for all the seperate devices isn't always the right thing. These devices include anything that the targets are using, exception stores for snapshots, logs for mirror, all paths for multipath (or'ing is most likely to be wrong for multipath).

Yeah the patch is pretty much crap in that area, I don't think Miquel was aiming for inclusion :)

I'd suggest making queue functions for congestion state as well so it stacks properly.

The pdflush-is-running-on-this-queue bit can probably remain as-is. It's
mostly meant to prevent 2 pdflush daemons from running the same queue. I don't see much harm in pdflush #1 running on /dev/md0 which consists of /dev/sda1 and /dev/sdb1 and pdflush #2 running on /dev/sdb2, right ?

Quoted 10 lines

Quoted 3 lines

Even if it isn't happening
a lot, and something isn't bust it might be a good idea to do this.

Seems OK from a quick check. pdflush will block in get_request_wait() occasionally, but not at all often. Perhaps we could move the write_congested test into the mpage_writepages() inner loop but it hardly seems worth the risk.

Maybe things are different on Miquel's clockwork controller.

I haven't tested it yet because of the "This patch isn't actually so good" comment, but I found another explanation.

Quoted 3 lines

 drivers/block/ll_rw_blk.c |    2 ++
 fs/fs-writeback.c         |    2 ++
 2 files changed, 4 insertions(+)

*Lightbulb on* .. I just read fs-writeback.c. As I said, this happens with an LVM device. Could it be that because LVM and the actual device have different struct request_queue's things go awry ?

In fs-writeback.c, your're looking at the LVM device (and its request_queue, and its backing_dev_info). In__make_request, you're looking at the SCSI device.

Quoted 18 lines · Miquel van Smoorenburg

Quoted 10 lines

Quoted 3 lines

Even if it isn't happening
a lot, and something isn't bust it might be a good idea to do this.

Seems OK from a quick check. pdflush will block in get_request_wait() occasionally, but not at all often. Perhaps we could move the write_congested test into the mpage_writepages() inner loop but it hardly seems worth the risk.

Maybe things are different on Miquel's clockwork controller.

I haven't tested it yet because of the "This patch isn't actually so good" comment, but I found another explanation.

Quoted 3 lines

 drivers/block/ll_rw_blk.c |    2 ++
 fs/fs-writeback.c         |    2 ++
 2 files changed, 4 insertions(+)

Quoted 7 lines

*Lightbulb on* .. I just read fs-writeback.c. As I said, this happens with an LVM device. Could it be that because LVM and the actual device have different struct request_queue's things go awry ?

In fs-writeback.c, your're looking at the LVM device (and its request_queue, and its backing_dev_info). In__make_request, you're looking at the SCSI device.

In principle, the lvm/md queues themselves will never be congested. But the underlying queues can be, of course.

Now this approach is _much_ better, imo. I don't particularly care very much for how you solved it, though, I'd much rather just see both setting and testing passed down (and kill the ->aux as well).

Hi Miquel,
Can you see if this patch helps you?

No, I'm actually referring to a struct request. I'm logging this in the SCSI layer, in scsi_request_fn(), just after elv_next_request(). I have in fact logged all the bio's submitted to __make_request, and the output of the elevator from elv_next_request(). The bio's are submitted sequentially, the resulting requests aren't. But this is because nr_requests is 128, while the 3ware device has a queue of 254 entries (no tagging though). Upping nr_requests to 512 makes this go away ..

That shouldn't be necessary though. I only see this with LVM over 3ware-raid5, not on the 3ware-raid5 array directly (/dev/sda1). And it gets less troublesome with a lot of debugging (unless I set nr_requests lower again), which points to a timing issue.

Quoted 11 lines · Miquel van Smoorenburg

No, I'm actually referring to a struct request. I'm logging this in the SCSI layer, in scsi_request_fn(), just after elv_next_request(). I have in fact logged all the bio's submitted to __make_request, and the output of the elevator from elv_next_request(). The bio's are submitted sequentially,
the resulting requests aren't. But this is because nr_requests is 128, while
the 3ware device has a queue of 254 entries (no tagging though). Upping nr_requests to 512 makes this go away ..

Quoted 9 lines

That shouldn't be necessary though. I only see this with LVM over 3ware-raid5,
not on the 3ware-raid5 array directly (/dev/sda1). And it gets less troublesome
with a lot of debugging (unless I set nr_requests lower again), which points
to a timing issue.

So the problem you are seeing is due to "unlucky" timing between two processes submitting IO. And the very efficient mechanisms (merging, sorting) we have to improve situations exactly like this is effectively disabled. And to make it worse, it appears that your controller shits itself on this trivially simple pattern.

Your hack makes a baby step in the direction of per *process* request limits, which I happen to be an advocate of. As it stands though, I don't like it.

Quoted 4 lines · Miquel van Smoorenburg

For some reason, when using LVM, write requests get queued out of order to the 3ware controller, which results in quite a bit of seeking and thus performance loss.

[..]

Quoted 18 lines

Okay I repeated some earlier tests, and I added some debug code in several places.

I added logging to tw_scsi_queue() in the 3ware driver to log the start sector and length of each request. It logs something like:
3wdbg: id 119, lba = 0x2330bc33, num_sectors = 256

With a perl script, I can check if the requests are sent to the host in order. That outputs something like this:

Consecutive: start 1180906348, length 7936 sec (3968 KB), requests: 31
Consecutive: start 1180906340, length 8 sec (4 KB), requests: 1
Consecutive: start 1180914292, length 7936 sec (3968 KB), requests: 31
Consecutive: start 1180914284, length 8 sec (4 KB), requests: 1
Consecutive: start 1180922236, length 7936 sec (3968 KB), requests: 31
Consecutive: start 1180922228, length 8 sec (4 KB), requests: 1
Consecutive: start 1180930180, length 7936 sec (3968 KB), requests: 31

Quoted 2 lines

See, 31 requests in order, then one request "backwards", then 31 in order, etc.

I found out what causes this. It's get_request_wait().

When the request queue is full, and a new request needs to be created, __make_request() blocks in get_request_wait().

Another process wakes up first (pdflush / process submitting I/O itself / xfsdatad / etc) and sends the next bio's to __make_request(). In the mean time some free requests have become available, and the bios are merged into a new request. Those requests are submitted to the device.

Then, get_request_wait() returns but the bio is not mergeable anymore - and that results in a backwards seek, severely limiting the I/O rate.

Wouldn't it be better to allow the request allocation and queue the request, and /then/ put the process to sleep ? The queue will grow larger than nr_requests, but it does that anyway.

Quoted 5 lines · Miquel van Smoorenburg

I found out what causes this. It's get_request_wait().

When the request queue is full, and a new request needs to be created, __make_request() blocks in get_request_wait().

Quoted 8 lines

Another process wakes up first (pdflush / process submitting I/O itself / xfsdatad / etc) and sends the next bio's to __make_request(). In the mean time some free requests have become available, and the bios are merged into a new request. Those requests are submitted to the device.

Then, get_request_wait() returns but the bio is not mergeable anymore - and that results in a backwards seek, severely limiting the I/O rate.

The "batching" logic there should allow a process to submit a number of requests even above the nr_requests limit to prevent this interleave and context switching.

Are you using tagged command queueing? What depth?

Quoted 21 lines · Nick Piggin

Quoted 5 lines · Miquel van Smoorenburg

I found out what causes this. It's get_request_wait().

When the request queue is full, and a new request needs to be created, __make_request() blocks in get_request_wait().

Quoted 9 lines

Another process wakes up first (pdflush / process submitting I/O itself / xfsdatad / etc) and sends the next bio's to __make_request(). In the mean time some free requests have become available, and the bios are merged into a new request. Those requests are submitted to the device.

Then, get_request_wait() returns but the bio is not mergeable anymore - and that results in a backwards seek, severely limiting the I/O rate.

The "batching" logic there should allow a process to submit a number of requests even above the nr_requests limit to prevent this interleave and context switching.

Quoted 2 lines

Are you using tagged command queueing? What depth?

No, I'm not using tagged command queueing. The 3ware controller is not a real scsi controller, the driver just emulates one. It's a raid5 controller that drives SATA disks. It has an internal request queue ("can_queu") of 254 outstanding commands.

Because that is way bigger than nr_requests
this happens - if I set nr_requests to 512, the problem goes away. But that shouldn't happen ;)

I'm preparing a proof-of-concept patch now, if it works and I don't wedge the remote machine I'm testing this on I'll post it in a few minutes.