block: Change default IO scheduler to deadline except SATA

Hi,

I am wondering if CFQ as default scheduler is still the right choice. CFQ generally works well on slow rotational media (SATA?). But often underperforms on faster storage (storage arrays, PCIE SSDs, virtualized disk in linux guests etc). People often put logic in user space to tune their systems and change IO scheduler to deadline to get better performance on faster storage.

Though there is not one good answer for all kind of storage and for all kind of workloads, I am wondering if we can provide a better default and that is change default IO scheduler to "deadline" except SATA.

One can argue that some SAS disks can be slow too and benefit from CFQ. Yes, but default IO scheduler choice is not perfect anyway. It just tries to cater to a wide variety of use cases out of the box.

So I am throwing this patch out see if it flies. Personally, I think it might turn out to be a more reasonable default.

Quoted 17 lines · Vivek Goyal

Hi,

I am wondering if CFQ as default scheduler is still the right choice. CFQ generally works well on slow rotational media (SATA?). But often underperforms on faster storage (storage arrays, PCIE SSDs, virtualized disk in linux guests etc). People often put logic in user space to tune their systems and change IO scheduler to deadline to get better performance on faster storage.

Though there is not one good answer for all kind of storage and for all kind of workloads, I am wondering if we can provide a better default and that is change default IO scheduler to "deadline" except SATA.

One can argue that some SAS disks can be slow too and benefit from CFQ. Yes, but default IO scheduler choice is not perfect anyway. It just tries to cater to a wide variety of use cases out of the box.

Quoted 2 lines

So I am throwing this patch out see if it flies. Personally, I think it might turn out to be a more reasonable default.

I think it'd be a lot more sane to just use CFQ on rotational single devices, and default to deadline on raid or non-rotational devices. This still isn't perfect, since less worthy SSDs still benefit from the read/write separation, and some multi device configs will be faster as well. But it's better.

The below patch is not a good idea. There's no clear distinction between on what CFQ is now the default.

Quoted 18 lines · Jens Axboe

Quoted 17 lines · Vivek Goyal

Hi,

I am wondering if CFQ as default scheduler is still the right choice. CFQ generally works well on slow rotational media (SATA?). But often underperforms on faster storage (storage arrays, PCIE SSDs, virtualized disk in linux guests etc). People often put logic in user space to tune their systems and change IO scheduler to deadline to get better performance on faster storage.

Though there is not one good answer for all kind of storage and for all kind of workloads, I am wondering if we can provide a better default and that is change default IO scheduler to "deadline" except SATA.

One can argue that some SAS disks can be slow too and benefit from CFQ. Yes, but default IO scheduler choice is not perfect anyway. It just tries to cater to a wide variety of use cases out of the box.

Quoted 8 lines

Quoted 2 lines

So I am throwing this patch out see if it flies. Personally, I think it might turn out to be a more reasonable default.

I think it'd be a lot more sane to just use CFQ on rotational single devices, and default to deadline on raid or non-rotational devices. This still isn't perfect, since less worthy SSDs still benefit from the read/write separation, and some multi device configs will be faster as well. But it's better.

Hi Jens,

Thanks. Taking a decision based on rotational flag makes sense. I am not sure that does one get the information that a block device is a single device or not. Especially with HBAs, SCSI Luns over Fiber, iSCSI Luns etc. I have few Scsi Luns exported to me backed by a storage array. Everything runs CFQ by default. And though disks in the array are rotational, they are RAIDed and AFAIK, this information is not available to driver.

I am not sure if there is an easy way to get similar info for dm/md devices.

Quoted 21 lines · Vivek Goyal

Quoted 6 lines · Jens Axboe

I think it'd be a lot more sane to just use CFQ on rotational single devices, and default to deadline on raid or non-rotational devices. This
still isn't perfect, since less worthy SSDs still benefit from the read/write separation, and some multi device configs will be faster as well. But it's better.

Hi Jens,

Thanks. Taking a decision based on rotational flag makes sense. I am not sure that does one get the information that a block device is a single
device or not. Especially with HBAs, SCSI Luns over Fiber, iSCSI Luns etc.
I have few Scsi Luns exported to me backed by a storage array. Everything
runs CFQ by default. And though disks in the array are rotational, they are RAIDed and AFAIK, this information is not available to driver.

Quoted 2 lines

I am not sure if there is an easy way to get similar info for dm/md devices.

Thinking more about it, even if we have a way to define a request queue flag for multi devices (QUEUE_FLAG_MULTI_DEVICE), when can block layer take a decision to change the IO scheduler. At queue alloc and init time driver might not have even called add_disk() or set all the flags/properties of the queue. So doing it at queue alloc/init time might not be best.

And later we get control only when actual IO happens on the queue and doing one more check or trying to change elevator in IO path is not a good idea.

May be when driver tries to set ROTATIONAL or MULTI_DEVICE flag, we can check and change elevator then.

So we are back to the question of can scsi devices find out if a Lun is backed by single disk or multiple disks.

Quoted 2 lines

So we are back to the question of can scsi devices find out if a Lun is backed by single disk or multiple disks.

I'm not aware of any discrete attribute (comparable to 'rotational' flag) that SCSI devices will advertise that indicates "I'm a raid array".

That said, we can have a _very_ good hint that a SCSI device is a raid array if:

1) optimal_io_size is not zero, minimum_io_size is not equal to optimal_io_size, and optimal_io_size is a multiple of minimum_io_size

2) WCE=0 (higher-end arrays with a writeback cache)

Determining 1 could be enough, we should probably ignore 2 as it isn't an absolute indication that a device is composed of multiple devices (especially not if considered independently of 1).

On Tue, Apr 10, 2012 at 12:13:07PM -0400, Mike Snitzer wrote:

[..]

Quoted 9 lines

Quoted 2 lines

So we are back to the question of can scsi devices find out if a Lun is backed by single disk or multiple disks.

I'm not aware of any discrete attribute (comparable to 'rotational' flag) that SCSI devices will advertise that indicates "I'm a raid array".

That said, we can have a _very_ good hint that a SCSI device is a raid array if:

Quoted 8 lines

1) optimal_io_size is not zero, minimum_io_size is not equal to optimal_io_size, and optimal_io_size is a multiple of minimum_io_size

2) WCE=0 (higher-end arrays with a writeback cache)

Determining 1 could be enough, we should probably ignore 2 as it isn't an absolute indication that a device is composed of multiple devices (especially not if considered independently of 1).

Umm..., somehow relying on optimal_io_size != minimum_io_size sounds odd to me (assuming it works).

I checked bunch of Luns exported to me and all of them have optimal_io_size=0.

I have few FC Luns exported from two array vendors. I have few iscsi Luns exported from two separate vendors and all these luns have optimal_io_size=0.

That said, we can have a _very_ good hint that a SCSI device is a raid array if:

1) optimal_io_size is not zero, minimum_io_size is not equal to optimal_io_size, and optimal_io_size is a multiple of minimum_io_size

2) WCE=0 (higher-end arrays with a writeback cache)

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

"Mike" == Mike Snitzer <[email protected]> writes:

Quoted 30 lines · Vivek Goyal

Quoted 18 lines · Jens Axboe

Quoted 17 lines · Vivek Goyal

Hi,

I am wondering if CFQ as default scheduler is still the right choice. CFQ generally works well on slow rotational media (SATA?). But often underperforms on faster storage (storage arrays, PCIE SSDs, virtualized disk in linux guests etc). People often put logic in user space to tune their systems and change IO scheduler to deadline to get better performance on faster storage.

Though there is not one good answer for all kind of storage and for all kind of workloads, I am wondering if we can provide a better default and that is change default IO scheduler to "deadline" except SATA.

One can argue that some SAS disks can be slow too and benefit from CFQ. Yes, but default IO scheduler choice is not perfect anyway. It just tries to cater to a wide variety of use cases out of the box.

Quoted 8 lines

Quoted 2 lines

So I am throwing this patch out see if it flies. Personally, I think it might turn out to be a more reasonable default.

I think it'd be a lot more sane to just use CFQ on rotational single devices, and default to deadline on raid or non-rotational devices. This still isn't perfect, since less worthy SSDs still benefit from the read/write separation, and some multi device configs will be faster as well. But it's better.

Hi Jens,

Quoted 8 lines

Thanks. Taking a decision based on rotational flag makes sense. I am not sure that does one get the information that a block device is a single device or not. Especially with HBAs, SCSI Luns over Fiber, iSCSI Luns etc. I have few Scsi Luns exported to me backed by a storage array. Everything runs CFQ by default. And though disks in the array are rotational, they are RAIDed and AFAIK, this information is not available to driver.

I am not sure if there is an easy way to get similar info for dm/md devices.

Thinking more about it, even if we have a way to define a request queue flag for multi devices (QUEUE_FLAG_MULTI_DEVICE), when can block layer take a decision to change the IO scheduler. At queue alloc and init time driver might not have even called add_disk() or set all the flags/properties of the queue. So doing it at queue alloc/init time might not be best.

And later we get control only when actual IO happens on the queue and doing one more check or trying to change elevator in IO path is not a good idea.

May be when driver tries to set ROTATIONAL or MULTI_DEVICE flag, we can check and change elevator then.

So we are back to the question of can scsi devices find out if a Lun is backed by single disk or multiple disks.

Quoted 2 lines

So we are back to the question of can scsi devices find out if a Lun is backed by single disk or multiple disks.

The cleanest would be to have the driver signal these attributes at probe time. You could even adjust CFQ properties based on this, driving the queue depth harder etc. Realistically, going forward, most fast flash devices will be driven by a noop-like scheduler on multiqueue. So CPU cost of the IO scheduler can mostly be ignored, since CFQ cost on even big RAIDs isn't an issue due to the low IOPS rates.

On Tue, Apr 10, 2012 at 08:41:08PM +0200, Jens Axboe wrote:

[..]

Quoted 9 lines

Quoted 2 lines

So we are back to the question of can scsi devices find out if a Lun is backed by single disk or multiple disks.

The cleanest would be to have the driver signal these attributes at probe time. You could even adjust CFQ properties based on this, driving the queue depth harder etc. Realistically, going forward, most fast flash devices will be driven by a noop-like scheduler on multiqueue. So CPU cost of the IO scheduler can mostly be ignored, since CFQ cost on even big RAIDs isn't an issue due to the low IOPS rates.

Agreed that on RAID CPU cost is not a problem. Just that idling and low queue depth kills the performance.

So apart from "rotational" if driver can give some hints about underlying devices being RAID (or multi device), it will help. Just that it looks like scsi does not have a way to determine that.

Quoted 13 lines · Vivek Goyal

On Tue, Apr 10, 2012 at 08:41:08PM +0200, Jens Axboe wrote:

[..]

Quoted 9 lines

Quoted 2 lines

So we are back to the question of can scsi devices find out if a Lun is backed by single disk or multiple disks.

The cleanest would be to have the driver signal these attributes at probe time. You could even adjust CFQ properties based on this, driving the queue depth harder etc. Realistically, going forward, most fast flash devices will be driven by a noop-like scheduler on multiqueue. So CPU cost of the IO scheduler can mostly be ignored, since CFQ cost on even big RAIDs isn't an issue due to the low IOPS rates.

Quoted 2 lines

Agreed that on RAID CPU cost is not a problem. Just that idling and low queue depth kills the performance.

Exactly, and both of these are trivially adjustable as long as we know when to do it.

Quoted 3 lines

So apart from "rotational" if driver can give some hints about underlying devices being RAID (or multi device), it will help. Just that it looks like scsi does not have a way to determine that.

This sort of thing should be done with a udev rule.

It should not be too
hard to match for the most popular arrays, catching the majority of the setups by default. Or you could ask the SCSI folks for some heuristics, it's not unlikely that a few different attributes could make that bullet proof, pretty much.

Quoted 17 lines · Jens Axboe

Quoted 13 lines · Vivek Goyal

On Tue, Apr 10, 2012 at 08:41:08PM +0200, Jens Axboe wrote:

[..]

Quoted 9 lines

Quoted 2 lines

So we are back to the question of can scsi devices find out if a Lun is backed by single disk or multiple disks.

The cleanest would be to have the driver signal these attributes at probe time. You could even adjust CFQ properties based on this, driving the queue depth harder etc. Realistically, going forward, most fast flash devices will be driven by a noop-like scheduler on multiqueue. So CPU cost of the IO scheduler can mostly be ignored, since CFQ cost on even big RAIDs isn't an issue due to the low IOPS rates.

Quoted 2 lines

Agreed that on RAID CPU cost is not a problem. Just that idling and low queue depth kills the performance.

Quoted 8 lines

Exactly, and both of these are trivially adjustable as long as we know when to do it.

Quoted 3 lines

So apart from "rotational" if driver can give some hints about underlying devices being RAID (or multi device), it will help. Just that it looks like scsi does not have a way to determine that.

This sort of thing should be done with a udev rule.

[CCing kay]

Kay does not like the idea of doing something along this line in udev. He thinks that kernel changes over a period of time making udev rules stale and hence it should be done in kernel. :-) I think he has had some not so good experiences in the past.

Though personally I think that anything which is not set in stone should go to udev. It atleast allows for easy change if user does not like the setting. (disable the rule, modify the rule etc). And then rules evolve as things change in kernel.

Anyway, this point can be detabted later once we figure out what's the set of atrributes to look at.

Quoted 5 lines

It should not be too
hard to match for the most popular arrays, catching the majority of the setups by default. Or you could ask the SCSI folks for some heuristics, it's not unlikely that a few different attributes could make that bullet proof, pretty much.

I am wondering what will happen to request based multipath targets in this scheme. There will have to be I guess additional rules to look for underlying paths and then change the io scheduler accordingly.

Quoted 34 lines · Vivek Goyal

Quoted 17 lines · Jens Axboe

Quoted 13 lines · Vivek Goyal

On Tue, Apr 10, 2012 at 08:41:08PM +0200, Jens Axboe wrote:

[..]

Quoted 9 lines

Quoted 2 lines

So we are back to the question of can scsi devices find out if a Lun is backed by single disk or multiple disks.

The cleanest would be to have the driver signal these attributes at probe time. You could even adjust CFQ properties based on this, driving the queue depth harder etc. Realistically, going forward, most fast flash devices will be driven by a noop-like scheduler on multiqueue. So CPU cost of the IO scheduler can mostly be ignored, since CFQ cost on even big RAIDs isn't an issue due to the low IOPS rates.

Quoted 2 lines

Agreed that on RAID CPU cost is not a problem. Just that idling and low queue depth kills the performance.

Quoted 8 lines

Exactly, and both of these are trivially adjustable as long as we know when to do it.

Quoted 3 lines

So apart from "rotational" if driver can give some hints about underlying devices being RAID (or multi device), it will help. Just that it looks like scsi does not have a way to determine that.

This sort of thing should be done with a udev rule.

[CCing kay]

Kay does not like the idea of doing something along this line in udev. He thinks that kernel changes over a period of time making udev rules stale and hence it should be done in kernel. :-) I think he has had some not so good experiences in the past.

Quoted 7 lines

Though personally I think that anything which is not set in stone should go to udev. It atleast allows for easy change if user does not like the setting. (disable the rule, modify the rule etc). And then rules evolve as things change in kernel.

Anyway, this point can be detabted later once we figure out what's the set of atrributes to look at.

It's a bit tricky. But supposedly sysfs files are part of the ABI, no matter how silly that may be. For these particular tunables, that means that some parts of the ABI are only valid/there if others contain a specific value. So I'm assuming that udev does not want to rely on that. Now I don't know a lot about udev or udev rules, but if you could make it depend on the value of <dev>/queue/scheduler, then it should (supposedly) be stable and safe to rely on. It all depends on what kind of logic you can stuff into the rules.

In any case, I'm sure that udev does not want to ship with those rules. It would have to be a separate package. Which is fine, in my opinion.

Quoted 6 lines

Quoted 5 lines

It should not be too
hard to match for the most popular arrays, catching the majority of the setups by default. Or you could ask the SCSI folks for some heuristics, it's not unlikely that a few different attributes could make that bullet proof, pretty much.

Quoted 3 lines

I am wondering what will happen to request based multipath targets in this scheme. There will have to be I guess additional rules to look for underlying paths and then change the io scheduler accordingly.

If each path is a device, each device should get caught and matched.

Quoted 61 lines · Jens Axboe

Quoted 34 lines · Vivek Goyal

Quoted 17 lines · Jens Axboe

Quoted 13 lines · Vivek Goyal

On Tue, Apr 10, 2012 at 08:41:08PM +0200, Jens Axboe wrote:

[..]

Quoted 9 lines

Quoted 2 lines

So we are back to the question of can scsi devices find out if a Lun is backed by single disk or multiple disks.

The cleanest would be to have the driver signal these attributes at probe time. You could even adjust CFQ properties based on this, driving the queue depth harder etc. Realistically, going forward, most fast flash devices will be driven by a noop-like scheduler on multiqueue. So CPU cost of the IO scheduler can mostly be ignored, since CFQ cost on even big RAIDs isn't an issue due to the low IOPS rates.

Quoted 2 lines

Agreed that on RAID CPU cost is not a problem. Just that idling and low queue depth kills the performance.

Quoted 8 lines

Exactly, and both of these are trivially adjustable as long as we know when to do it.

Quoted 3 lines

So apart from "rotational" if driver can give some hints about underlying devices being RAID (or multi device), it will help. Just that it looks like scsi does not have a way to determine that.

This sort of thing should be done with a udev rule.

[CCing kay]

Kay does not like the idea of doing something along this line in udev. He thinks that kernel changes over a period of time making udev rules stale and hence it should be done in kernel. :-) I think he has had some not so good experiences in the past.

Quoted 7 lines

Though personally I think that anything which is not set in stone should go to udev. It atleast allows for easy change if user does not like the setting. (disable the rule, modify the rule etc). And then rules evolve as things change in kernel.

Anyway, this point can be detabted later once we figure out what's the set of atrributes to look at.

It's a bit tricky. But supposedly sysfs files are part of the ABI, no matter how silly that may be. For these particular tunables, that means that some parts of the ABI are only valid/there if others contain a specific value. So I'm assuming that udev does not want to rely on that. Now I don't know a lot about udev or udev rules, but if you could make it depend on the value of <dev>/queue/scheduler, then it should (supposedly) be stable and safe to rely on. It all depends on what kind of logic you can stuff into the rules.

In any case, I'm sure that udev does not want to ship with those rules. It would have to be a separate package. Which is fine, in my opinion.

Quoted 6 lines

Quoted 5 lines

It should not be too
hard to match for the most popular arrays, catching the majority of the setups by default. Or you could ask the SCSI folks for some heuristics, it's not unlikely that a few different attributes could make that bullet proof, pretty much.

Quoted 5 lines

Quoted 3 lines

I am wondering what will happen to request based multipath targets in this scheme. There will have to be I guess additional rules to look for underlying paths and then change the io scheduler accordingly.

If each path is a device, each device should get caught and matched.

I'm still missing your position (other than you now wanting to make it a userspace concern).

Put differently: why should CFQ still be the default?

It is pretty widely held that deadline is the more sane default (multiple distros are now using it, deadline is default for guests, etc). CFQ has become more niche. The Linux default really should reflect this.

The only case where defaulting to CFQ seems to make sense is rotational SATA (and USB).

Quoted 5 lines · Mike Snitzer

I'm still missing your position (other than you now wanting to make it a userspace concern).

Put differently: why should CFQ still be the default?

Quoted 7 lines

It is pretty widely held that deadline is the more sane default (multiple distros are now using it, deadline is default for guests, etc). CFQ has become more niche. The Linux default really should reflect this.

The only case where defaulting to CFQ seems to make sense is rotational SATA (and USB).

That's the precisely the reason it should still be the default. The default settings should reflect a good user experience out of the box. Most desktop machines are still using SATA drives. And even those that made the leap to SSD, lots of those are still pretty sucky at high queue depths or without read/write separation. So I'm quite sure the default still makes a lot of sense.

Punt tuning to the server side. If you absolutely want the best performance out of your _particular_ workload, you are expecting and required to tune things anyway. Not just the IO scheduler, but in general. You can't make the same requirements for the desktop.

As to kernel vs user, I just see little reason for doing it in the kernel if we can put that policy in user space.