block-throttle: add .high limit

Hi,

The background is we don't have an ioscheduler for blk-mq yet, so we can't prioritize processes/cgroups.

This patch set tries to add basic arbitration
between cgroups with blk-throttle. It adds a new limit io.high for blk-throttle. It's only for cgroup2.

io.max is a hard limit throttling. cgroups with a max limit never dispatch more IO than their max limit. While io.high is a best effort throttling. cgroups with high limit can run above their high limit at appropriate time. Specifically, if all cgroups reach their high limit, all cgroups can run above their high limit. If any cgroup runs under its high limit, all other cgroups will run according to their high limit.

Quoted 4 lines · Shaohua Li

Hi,

The background is we don't have an ioscheduler for blk-mq yet, so we can't prioritize processes/cgroups.

So this is an interim solution till we have ioscheduler for blk-mq?

Quoted 10 lines

This patch set tries to add basic arbitration
between cgroups with blk-throttle. It adds a new limit io.high for blk-throttle. It's only for cgroup2.

io.max is a hard limit throttling. cgroups with a max limit never dispatch more IO than their max limit. While io.high is a best effort throttling. cgroups with high limit can run above their high limit at appropriate time. Specifically, if all cgroups reach their high limit, all cgroups can run above their high limit. If any cgroup runs under its high limit, all other cgroups will run according to their high limit.

Hi Shaohua,

I still don't understand why we should not implement a weight based proportional IO mechanism and how this mechanism is better than proportional IO .

Agreed that we have issues with proportional IO and we don't have good solutions for these problems. But I can't see that how this mechanism will overcome these problems either.

IIRC, biggest issue with proportional IO was that a low prio group might fill up the device queue with plenty of IO requests and later when high prio cgroup comes, it will still experience latencies anyway. And solution to the problem probably would be to get some awareness in device about priority of request and map weights to those priority. That way higher prio requests get prioritized.

Or run device at lower queue depth. That will improve latencies but migth reduce overall throughput.

Or thorottle number of buffered writes (as Jens's writeback throttling) patches were doing. Buffered writes seem to be biggest culprit for increased latencies and being able to control these should help.

ioprio/weight based proportional IO mechanism is much more generic and much easier to configure for any kind of storage. io.high is absolute limit and makes it much harder to configure. One needs to know a lot about underlying volume/device's bandwidth (which varies a lot anyway based on workload).

IMHO, we seem to be trying to cater to one specific use case using this mechanism. Something ioprio/weight based will be much more generic and we should explore implementing that along with building notion of ioprio in devices. When these two work together, we might be able to see good results. Just software mechanism alone might not be enough.

Il giorno 04 ott 2016, alle ore 17:56, Tejun Heo <[email protected]> ha scritto:

Hello, Vivek.

Quoted 7 lines · Vivek Goyal

Quoted 4 lines · Shaohua Li

Hi,

The background is we don't have an ioscheduler for blk-mq yet, so we can't prioritize processes/cgroups.

So this is an interim solution till we have ioscheduler for blk-mq?

It's a common permanent solution which applies to both !mq and mq.

Quoted 13 lines

Quoted 10 lines

This patch set tries to add basic arbitration
between cgroups with blk-throttle. It adds a new limit io.high for blk-throttle. It's only for cgroup2.

io.max is a hard limit throttling. cgroups with a max limit never dispatch more IO than their max limit. While io.high is a best effort throttling. cgroups with high limit can run above their high limit at appropriate time. Specifically, if all cgroups reach their high limit, all cgroups can run above their high limit. If any cgroup runs under its high limit, all other cgroups will run according to their high limit.

Hi Shaohua,

Quoted 2 lines

I still don't understand why we should not implement a weight based proportional IO mechanism and how this mechanism is better than proportional IO .

Oh, if we actually can implement proportional IO control, it'd be great. The problem is that we have no way of knowing IO cost for highspeed ssd devices. CFQ gets around the problem by using the walltime as the measure of resource usage and scheduling time slices, which works fine for rotating disks but horribly for highspeed ssds.

We can get some semblance of proportional control by just counting bw or iops but both break down badly as a means to measure the actual resource consumption depending on the workload. While limit based control is more tedious to configure, it doesn't misrepresent what's going on and is a lot less likely to produce surprising outcomes.

We *can* try to concoct something which tries to do proportional control for highspeed ssds but that's gonna be quite a bit of complexity and I'm not so sure it'd be justifiable given that we can't even figure out measurement of the most basic operating unit.

Quoted 3 lines

Agreed that we have issues with proportional IO and we don't have good solutions for these problems. But I can't see that how this mechanism will overcome these problems either.

It mostly defers the burden to the one who's configuring the limits and expects it to know the characteristics of the device and workloads and configure accordingly. It's quite a bit more tedious to use but should be able to cover good portion of use cases without being overly complicated. I agree that it'd be nice to have a simple proportional control but as you said can't see a good solution for it at the moment.

Quoted 6 lines

IIRC, biggest issue with proportional IO was that a low prio group might fill up the device queue with plenty of IO requests and later when high prio cgroup comes, it will still experience latencies anyway. And solution to the problem probably would be to get some awareness in device about priority of request and map weights to those priority. That way higher prio requests get prioritized.

Nah, the real problem is that we can't even decide what the proportions should be based on. The most fundamental part is missing.

Quoted 2 lines

Or run device at lower queue depth. That will improve latencies but migth reduce overall throughput.

And that we can't do this (and thus basically operate close to scheduling time slices) for highspeed ssds.

Quoted 3 lines

Or thorottle number of buffered writes (as Jens's writeback throttling) patches were doing. Buffered writes seem to be biggest culprit for increased latencies and being able to control these should help.

That's a different topic.

Quoted 5 lines

ioprio/weight based proportional IO mechanism is much more generic and much easier to configure for any kind of storage. io.high is absolute limit and makes it much harder to configure. One needs to know a lot about underlying volume/device's bandwidth (which varies a lot anyway based on workload).

Yeap, no disagreement there, but it still is a workable solution.

Quoted 6 lines

IMHO, we seem to be trying to cater to one specific use case using this mechanism. Something ioprio/weight based will be much more generic and we should explore implementing that along with building notion of ioprio in devices. When these two work together, we might be able to see good results. Just software mechanism alone might not be enough.

I don't think it's catering to specific use cases. It is a generic mechanism which demands knowledge and experimentation to configure. It's more a way for the kernel to cop out and defer figuring out device characteristics to userland. If you have a better idea, I'm all ears.

Thanks.

Signature · 5 lines

tejun
--
To unsubscribe from this list: send the line "unsubscribe linux-block" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Could you please elaborate more on this point? BFQ uses sectors served to measure service, and, on the all the fast devices on which we have tested it, it accurately distributes
bandwidth as desired, redistributes excess bandwidth with any issue, and guarantees high responsiveness and low latency at application and system level (e.g., ~0 drop rate in video playback, with any background workload tested).

Could you please suggest me some test to show how sector-based guarantees fails?

Il giorno 04 ott 2016, alle ore 18:27, Tejun Heo <[email protected]> ha scritto:

Hello,

Quoted 7 lines · Paolo Valente

Could you please elaborate more on this point? BFQ uses sectors served to measure service, and, on the all the fast devices on which we have tested it, it accurately distributes
bandwidth as desired, redistributes excess bandwidth with any issue, and guarantees high responsiveness and low latency at application and system level (e.g., ~0 drop rate in video playback, with any background workload tested).

The same argument as before. Bandwidth is a very bad measure of IO resources spent. For specific use cases (like desktop or whatever), this can work but not generally.

Quoted 2 lines

Could you please suggest me some test to show how sector-based guarantees fails?

Well, mix 4k random and sequential workloads and try to distribute the acteual IO resources.

Maybe the source of confusion is the fact that a simple sector-based, proportional share scheduler always distributes total bandwidth according to weights. The catch is the additional BFQ rule: random workloads get only time isolation, and are charged for full budgets, so as to not affect the schedule of quasi-sequential workloads. So, the correct claim for BFQ is that it distributes total bandwidth according to weights (only) when all competing workloads are quasi-sequential. If some workloads are random, then these workloads are just time scheduled. This does break proportional-share bandwidth distribution with mixed workloads, but, much more importantly, saves both total throughput and individual bandwidths of quasi-sequential workloads.

We could then check whether I did succeed in tuning timeouts and budgets so as to achieve the best tradeoffs. But this is probably a second-order problem as of now.

Thanks.

Signature · 1 line

tejun

Quoted 14 lines

Il giorno 04 ott 2016, alle ore 18:27, Tejun Heo <[email protected]> ha scritto:

Hello,

Quoted 7 lines · Paolo Valente

Could you please elaborate more on this point? BFQ uses sectors served to measure service, and, on the all the fast devices on which we have tested it, it accurately distributes
bandwidth as desired, redistributes excess bandwidth with any issue, and guarantees high responsiveness and low latency at application and system level (e.g., ~0 drop rate in video playback, with any background workload tested).

Quoted 4 lines

The same argument as before. Bandwidth is a very bad measure of IO resources spent. For specific use cases (like desktop or whatever), this can work but not generally.

Actually, we have already discussed this point, and IMHO the arguments that (apparently) convinced you that bandwidth is the most relevant service guarantee for I/O in desktops and the like, prove that bandwidth is the most important service guarantee in servers too.

Again, all the examples I can think of seem to confirm it:
. file hosting: a good service must guarantee reasonable read/write, i.e., download/upload, speeds to users
. file streaming: a good service must guarantee low drop rates, and this can be guaranteed only by guaranteeing bandwidth and latency . web hosting: high bandwidth and low latency needed here too . clouds: high bw and low latency needed to let, e.g., users of VMs enjoy high responsiveness and, for example, reasonable file-copy time
...

To put in yet another way, with packet I/O in, e.g., clouds, there are basically the same issues, and the main goal is again guaranteeing bandwidth and low latency among nodes.

Could you please provide a concrete server example (assuming we still agree about desktops), where I/O bandwidth does not matter while time does?

Quoted 6 lines

Quoted 2 lines

Could you please suggest me some test to show how sector-based guarantees fails?

Well, mix 4k random and sequential workloads and try to distribute the acteual IO resources.

If I'm not mistaken, we have already gone through this example too, and I thought we agreed on what service scheme worked best, again focusing only on desktops. To make a long story short(er), here is a snippet from one of our last exchanges.

----------

Quoted 16 lines · Paolo Valente

Maybe the source of confusion is the fact that a simple sector-based, proportional share scheduler always distributes total bandwidth according to weights. The catch is the additional BFQ rule: random workloads get only time isolation, and are charged for full budgets, so as to not affect the schedule of quasi-sequential workloads. So, the correct claim for BFQ is that it distributes total bandwidth according to weights (only) when all competing workloads are quasi-sequential. If some workloads are random, then these workloads are just time scheduled. This does break proportional-share bandwidth distribution with mixed workloads, but, much more importantly, saves both total throughput and individual bandwidths of quasi-sequential workloads.

We could then check whether I did succeed in tuning timeouts and budgets so as to achieve the best tradeoffs. But this is probably a second-order problem as of now.

Il giorno 04 ott 2016, alle ore 19:28, Shaohua Li <[email protected]> ha scritto:

Quoted 39 lines · Paolo Valente

Quoted 14 lines

Il giorno 04 ott 2016, alle ore 18:27, Tejun Heo <[email protected]> ha scritto:

Hello,

Quoted 7 lines · Paolo Valente

Could you please elaborate more on this point? BFQ uses sectors served to measure service, and, on the all the fast devices on which we have tested it, it accurately distributes
bandwidth as desired, redistributes excess bandwidth with any issue, and guarantees high responsiveness and low latency at application and system level (e.g., ~0 drop rate in video playback, with any background workload tested).

Quoted 4 lines

The same argument as before. Bandwidth is a very bad measure of IO resources spent. For specific use cases (like desktop or whatever), this can work but not generally.

Actually, we have already discussed this point, and IMHO the arguments that (apparently) convinced you that bandwidth is the most relevant service guarantee for I/O in desktops and the like, prove that bandwidth is the most important service guarantee in servers too.

Again, all the examples I can think of seem to confirm it:
. file hosting: a good service must guarantee reasonable read/write, i.e., download/upload, speeds to users
. file streaming: a good service must guarantee low drop rates, and this can be guaranteed only by guaranteeing bandwidth and latency . web hosting: high bandwidth and low latency needed here too . clouds: high bw and low latency needed to let, e.g., users of VMs enjoy high responsiveness and, for example, reasonable file-copy time
...

To put in yet another way, with packet I/O in, e.g., clouds, there are basically the same issues, and the main goal is again guaranteeing bandwidth and low latency among nodes.

Quoted 3 lines

Could you please provide a concrete server example (assuming we still agree about desktops), where I/O bandwidth does not matter while time does?

I don't think IO bandwidth does not matter. The problem is bandwidth can't measure IO cost. For example, you can't say 8k IO costs 2x IO resource than 4k IO.

Quoted 29 lines

Quoted 6 lines

Quoted 2 lines

Could you please suggest me some test to show how sector-based guarantees fails?

Well, mix 4k random and sequential workloads and try to distribute the acteual IO resources.

If I'm not mistaken, we have already gone through this example too, and I thought we agreed on what service scheme worked best, again focusing only on desktops. To make a long story short(er), here is a snippet from one of our last exchanges.

----------

Quoted 13 lines · Paolo Valente

Maybe the source of confusion is the fact that a simple sector-based, proportional share scheduler always distributes total bandwidth according to weights. The catch is the additional BFQ rule: random workloads get only time isolation, and are charged for full budgets, so as to not affect the schedule of quasi-sequential workloads. So, the correct claim for BFQ is that it distributes total bandwidth according to weights (only) when all competing workloads are quasi-sequential. If some workloads are random, then these workloads are just time scheduled. This does break proportional-share bandwidth distribution with mixed workloads, but, much more importantly, saves both total throughput and individual bandwidths of quasi-sequential workloads.

Quoted 3 lines

Quoted 3 lines

We could then check whether I did succeed in tuning timeouts and budgets so as to achieve the best tradeoffs. But this is probably a second-order problem as of now.

I don't see why random/sequential matters for SSD. what really matters is request size and IO depth. Time scheduling is skeptical too, as workloads can dispatch all IO within almost 0 time in high queue depth disks.

Thanks,
Shaohua

Quoted 47 lines

Il giorno 04 ott 2016, alle ore 19:28, Shaohua Li <[email protected]> ha scritto:

Quoted 42 lines · Paolo Valente

Quoted 14 lines

Il giorno 04 ott 2016, alle ore 18:27, Tejun Heo <[email protected]> ha scritto:

Hello,

Quoted 7 lines · Paolo Valente

Could you please elaborate more on this point? BFQ uses sectors served to measure service, and, on the all the fast devices on which we have tested it, it accurately distributes
bandwidth as desired, redistributes excess bandwidth with any issue, and guarantees high responsiveness and low latency at application and system level (e.g., ~0 drop rate in video playback, with any background workload tested).

Quoted 4 lines

The same argument as before. Bandwidth is a very bad measure of IO resources spent. For specific use cases (like desktop or whatever), this can work but not generally.

Actually, we have already discussed this point, and IMHO the arguments that (apparently) convinced you that bandwidth is the most relevant service guarantee for I/O in desktops and the like, prove that bandwidth is the most important service guarantee in servers too.

Again, all the examples I can think of seem to confirm it:
. file hosting: a good service must guarantee reasonable read/write, i.e., download/upload, speeds to users
. file streaming: a good service must guarantee low drop rates, and this can be guaranteed only by guaranteeing bandwidth and latency . web hosting: high bandwidth and low latency needed here too . clouds: high bw and low latency needed to let, e.g., users of VMs enjoy high responsiveness and, for example, reasonable file-copy time
...

To put in yet another way, with packet I/O in, e.g., clouds, there are basically the same issues, and the main goal is again guaranteeing bandwidth and low latency among nodes.

Could you please provide a concrete server example (assuming we still agree about desktops), where I/O bandwidth does not matter while time does?

Quoted 4 lines

I don't think IO bandwidth does not matter. The problem is bandwidth can't measure IO cost. For example, you can't say 8k IO costs 2x IO resource than 4k IO.

For what goal do you need to be able to say this, once you succeeded in guaranteeing bandwidth and low latency to each process/client/group/node/user?

Quoted 29 lines

Quoted 29 lines

Quoted 6 lines

Quoted 2 lines

Could you please suggest me some test to show how sector-based guarantees fails?

Well, mix 4k random and sequential workloads and try to distribute the acteual IO resources.

If I'm not mistaken, we have already gone through this example too, and I thought we agreed on what service scheme worked best, again focusing only on desktops. To make a long story short(er), here is a snippet from one of our last exchanges.

----------

Quoted 13 lines · Paolo Valente

Maybe the source of confusion is the fact that a simple sector-based, proportional share scheduler always distributes total bandwidth according to weights. The catch is the additional BFQ rule: random workloads get only time isolation, and are charged for full budgets, so as to not affect the schedule of quasi-sequential workloads. So, the correct claim for BFQ is that it distributes total bandwidth according to weights (only) when all competing workloads are quasi-sequential. If some workloads are random, then these workloads are just time scheduled. This does break proportional-share bandwidth distribution with mixed workloads, but, much more importantly, saves both total throughput and individual bandwidths of quasi-sequential workloads.

Quoted 8 lines

Quoted 3 lines

Quoted 3 lines

We could then check whether I did succeed in tuning timeouts and budgets so as to achieve the best tradeoffs. But this is probably a second-order problem as of now.

I don't see why random/sequential matters for SSD. what really matters is request size and IO depth. Time scheduling is skeptical too, as workloads can dispatch all IO within almost 0 time in high queue depth disks.

That's an orthogonal issue. If what matter is, e.g., size, then it is enough to replace "sequential I/O" with "large-request I/O". In case I have been too vague, here is an example: I mean that, e.g, in an I/O scheduler you replace the function that computes whether a queue is seeky based on request distance, with a function based on request size. And this is exactly what has been already done, for example, in CFQ:

	if (blk_queue_nonrot(cfqd->queue))
		cfqq->seek_history |= (n_sec < CFQQ_SECT_THR_NONROT);
	else
		cfqq->seek_history |= (sdist > CFQQ_SEEK_THR);

Il giorno 04 ott 2016, alle ore 20:28, Shaohua Li <[email protected]> ha scritto:

Quoted 52 lines · Paolo Valente

Quoted 51 lines

Il giorno 04 ott 2016, alle ore 19:28, Shaohua Li <[email protected]> ha scritto:

Quoted 35 lines · Paolo Valente

Quoted 14 lines

Il giorno 04 ott 2016, alle ore 18:27, Tejun Heo <[email protected]> ha scritto:

Hello,

Quoted 7 lines · Paolo Valente

Could you please elaborate more on this point? BFQ uses sectors served to measure service, and, on the all the fast devices on which we have tested it, it accurately distributes
bandwidth as desired, redistributes excess bandwidth with any issue, and guarantees high responsiveness and low latency at application and system level (e.g., ~0 drop rate in video playback, with any background workload tested).

Quoted 4 lines

The same argument as before. Bandwidth is a very bad measure of IO resources spent. For specific use cases (like desktop or whatever), this can work but not generally.

Actually, we have already discussed this point, and IMHO the arguments that (apparently) convinced you that bandwidth is the most relevant service guarantee for I/O in desktops and the like, prove that bandwidth is the most important service guarantee in servers too.

Again, all the examples I can think of seem to confirm it:
. file hosting: a good service must guarantee reasonable read/write, i.e., download/upload, speeds to users
. file streaming: a good service must guarantee low drop rates, and this can be guaranteed only by guaranteeing bandwidth and latency . web hosting: high bandwidth and low latency needed here too . clouds: high bw and low latency needed to let, e.g., users of VMs enjoy high responsiveness and, for example, reasonable file-copy time
...

Quoted 7 lines

To put in yet another way, with packet I/O in, e.g., clouds, there are basically the same issues, and the main goal is again guaranteeing bandwidth and low latency among nodes.

Could you please provide a concrete server example (assuming we still agree about desktops), where I/O bandwidth does not matter while time does?

I don't think IO bandwidth does not matter. The problem is bandwidth can't measure IO cost. For example, you can't say 8k IO costs 2x IO resource than 4k IO.

Quoted 3 lines

For what goal do you need to be able to say this, once you succeeded in guaranteeing bandwidth and low latency to each process/client/group/node/user?

I think we are discussing if bandwidth should be used to measure IO for propotional IO scheduling.

Since bandwidth can't measure the cost and you are
using it to do arbitration, you will either have low latency but unfair bandwidth, or fair bandwidth but some workloads have unexpected high latency. But it might be ok depending on the latency target (for example, you can set the latency target high, so low latency is guaranteed*) and workload characteristics. I think the bandwidth based proporional scheduling will only work for workloads disk isn't fully utilized.

Quoted 46 lines

Quoted 29 lines

Quoted 29 lines

Quoted 6 lines

Quoted 2 lines

Could you please suggest me some test to show how sector-based guarantees fails?

Well, mix 4k random and sequential workloads and try to distribute the acteual IO resources.

If I'm not mistaken, we have already gone through this example too, and I thought we agreed on what service scheme worked best, again focusing only on desktops. To make a long story short(er), here is a snippet from one of our last exchanges.

----------

Quoted 13 lines · Paolo Valente

Maybe the source of confusion is the fact that a simple sector-based, proportional share scheduler always distributes total bandwidth according to weights. The catch is the additional BFQ rule: random workloads get only time isolation, and are charged for full budgets, so as to not affect the schedule of quasi-sequential workloads. So, the correct claim for BFQ is that it distributes total bandwidth according to weights (only) when all competing workloads are quasi-sequential. If some workloads are random, then these workloads are just time scheduled. This does break proportional-share bandwidth distribution with mixed workloads, but, much more importantly, saves both total throughput and individual bandwidths of quasi-sequential workloads.

Quoted 8 lines

Quoted 3 lines

Quoted 3 lines

We could then check whether I did succeed in tuning timeouts and budgets so as to achieve the best tradeoffs. But this is probably a second-order problem as of now.

I don't see why random/sequential matters for SSD. what really matters is request size and IO depth. Time scheduling is skeptical too, as workloads can dispatch all IO within almost 0 time in high queue depth disks.

That's an orthogonal issue. If what matter is, e.g., size, then it is enough to replace "sequential I/O" with "large-request I/O". In case I have been too vague, here is an example: I mean that, e.g, in an I/O scheduler you replace the function that computes whether a queue is seeky based on request distance, with a function based on request size. And this is exactly what has been already done, for example, in CFQ:

Quoted 4 lines

	if (blk_queue_nonrot(cfqd->queue))
		cfqq->seek_history |= (n_sec < CFQQ_SECT_THR_NONROT);
	else
		cfqq->seek_history |= (sdist > CFQQ_SEEK_THR);

CFQ is known not fair for SSD especially high queue depth SSD, so this doesn't mean correctness.

And based on request size for idle detection (so let cfqq backlog the disk) isn't very good. iodepth 1 4k workload could be idle, but iodepth 128 4k workload likely isn't idle (and the workload can dispatch 128 requests in almost 0 time in high queue depth disk).

Thanks,
Shaohua

Quoted 3 lines

I don't think IO bandwidth does not matter. The problem is bandwidth can't measure IO cost. For example, you can't say 8k IO costs 2x IO resource than 4k IO.

For what goal do you need to be able to say this, once you succeeded in guaranteeing bandwidth and low latency to each process/client/group/node/user?

Il giorno 04 ott 2016, alle ore 20:54, Tejun Heo <[email protected]> ha scritto:

Hello, Paolo.

Quoted 4 lines · Paolo Valente

Quoted 3 lines

I don't think IO bandwidth does not matter. The problem is bandwidth can't measure IO cost. For example, you can't say 8k IO costs 2x IO resource than 4k IO.

Quoted 3 lines

For what goal do you need to be able to say this, once you succeeded in guaranteeing bandwidth and low latency to each process/client/group/node/user?

For resource partitioning mostly. It's not a single user or purpose use case. The same device gets shared across unrelated workloads and we need to guarantee differing levels of quality of service to each regardless of the specifics of workload.

We actually need to be able
to control IO resources.

Thanks.

Signature · 1 line

tejun

That's exactly what BFQ has succeeded in doing in all the tests devised so far. Can you give me a concrete example for which I can try with BFQ and with any other mechanism you deem better. If you are right, numbers will just make your point.

Il giorno 04 ott 2016, alle ore 21:14, Tejun Heo <[email protected]> ha scritto:

Hello, Paolo.

Quoted 4 lines · Paolo Valente

That's exactly what BFQ has succeeded in doing in all the tests devised so far. Can you give me a concrete example for which I can try with BFQ and with any other mechanism you deem better. If you are right, numbers will just make your point.

Hmm... I think we already discussed this but here's a really simple case. There are three unknown workloads A, B and C and we want to give A certain best-effort guarantees (let's say around 80% of the underlying device) whether A is sharing the device with B or C.

I get that bfq can be a good compromise on most desktop workloads and behave reasonably well for some server workloads with the slice expiration mechanism but it really isn't an IO resource partitioning mechanism.

Thanks.

Signature · 1 line

tejun

Quoted 4 lines

Hmm... I think we already discussed this but here's a really simple case. There are three unknown workloads A, B and C and we want to give A certain best-effort guarantees (let's say around 80% of the underlying device) whether A is sharing the device with B or C.

That's the same example that you proposed me in our previous discussion. For this example I showed you, with many boring numbers, that with BFQ you get the most accurate distribution of the resource.

If you have enough stamina, I can repeat them again. To save your

patience, here is a very brief summary. In a concrete use case, the unknown workloads turn into something like this: there will be a first time interval during which A happens to be, say, sequential, B happens to be, say, random and C happens to be, say, quasi-sequential. Then there will be a next time interval during which their characteristics change, and so on. It is easy (but boring, I acknowledge it) to show that, for each of these time intervals BFQ provides the best possible service in terms of fairness, bandwidth distribution, stability and so on. Why? Because of the elastic bandwidth-time scheduling of BFQ that we already discussed, and because BFQ is naturally accurate in redistributing aggregate throughput proportionally, when needed.

Quoted 4 lines

I get that bfq can be a good compromise on most desktop workloads and behave reasonably well for some server workloads with the slice expiration mechanism but it really isn't an IO resource partitioning mechanism.

Right. My argument is that BFQ enables you to give to each client the bandwidth and low-latency guarantees you want. And this IMO is way better than partitioning a resource and then getting unavoidable unfairness and high latency.

Il giorno 04 ott 2016, alle ore 22:27, Tejun Heo <[email protected]> ha scritto:

Hello, Paolo.

Quoted 8 lines · Paolo Valente

Quoted 4 lines

Hmm... I think we already discussed this but here's a really simple case. There are three unknown workloads A, B and C and we want to give A certain best-effort guarantees (let's say around 80% of the underlying device) whether A is sharing the device with B or C.

That's the same example that you proposed me in our previous discussion. For this example I showed you, with many boring numbers, that with BFQ you get the most accurate distribution of the resource.

Yes, it is about the same example and what I understood was that "accurate distribution of the resources" holds as long as the randomness is incidental (ie. due to layout on the filesystem and so on) with the slice expiration mechanism offsetting the actually random workloads.

Quoted 1 line

If you have enough stamina, I can repeat them again. To save your

I'll go back to the thread and re-read them.

Quoted 11 lines

patience, here is a very brief summary. In a concrete use case, the unknown workloads turn into something like this: there will be a first time interval during which A happens to be, say, sequential, B happens to be, say, random and C happens to be, say, quasi-sequential. Then there will be a next time interval during which their characteristics change, and so on. It is easy (but boring, I acknowledge it) to show that, for each of these time intervals BFQ provides the best possible service in terms of fairness, bandwidth distribution, stability and so on. Why? Because of the elastic bandwidth-time scheduling of BFQ that we already discussed, and because BFQ is naturally accurate in redistributing aggregate throughput proportionally, when needed.

Yeah, that's what I remember and for workload above certain level of randomness its time consumption is mapped to bw, right?

Quoted 9 lines

Quoted 4 lines

I get that bfq can be a good compromise on most desktop workloads and behave reasonably well for some server workloads with the slice expiration mechanism but it really isn't an IO resource partitioning mechanism.

Right. My argument is that BFQ enables you to give to each client the bandwidth and low-latency guarantees you want. And this IMO is way better than partitioning a resource and then getting unavoidable unfairness and high latency.

But that statement only holds while bw is the main thing to guarantee, no? The level of isolation that we're looking for here is fairly strict adherence to sub/few-milliseconds in terms of high percentile scheduling latency while within the configured bw/iops limits, not "overall this device is being used pretty well".

Thanks.

Signature · 5 lines

tejun
--
To unsubscribe from this list: send the line "unsubscribe linux-block" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Anyway, to avoid going on with trying speculations and arguments, let me retry with a practical proposal. BFQ is out there, free. Let's just test, measure and check whether we have already a solution to the problems you/we are still trying to solve in Linux.

Il giorno 05 ott 2016, alle ore 15:12, Vivek Goyal <[email protected]> ha scritto:

On Wed, Oct 05, 2016 at 02:37:00PM +0200, Paolo Valente wrote:

[..]

Quoted 4 lines

Anyway, to avoid going on with trying speculations and arguments, let me retry with a practical proposal. BFQ is out there, free. Let's just test, measure and check whether we have already a solution to the problems you/we are still trying to solve in Linux.

Hi Paolo,

Does BFQ implementaiton scale for fast storage devices using blk-mq interface. We will want to make sure that locking and other overhead of BFQ is very minimal so that overall throughput does not suffer.

Vivek

In this respect, for your generic, unpredictable scenario to make sense, there must exist at least one real system that meets the requirements of such a scenario. Or, if such a real system does not yet exist, it must be possible to emulate it. If it is impossible to achieve this last goal either, then I miss the usefulness of looking for solutions for such a scenario.

That said, let's define the instance(s) of the scenario that you find most representative, and let's test BFQ on it/them. Numbers will give us the answers. For example, what about all or part of the following groups:
. one cyclically doing random I/O for some second and then sequential I/O for the next seconds
. one doing, say, quasi-sequential I/O in ON/OFF cycles
. one starting an application cyclically
. one playing back or streaming a movie

For each group, we could then measure the time needed to complete each phase of I/O in each cycle, plus the responsiveness in the group starting an application, plus the frame drop in the group streaming the movie. In addition, we can measure the bandwidth/iops enjoyed by each group, plus, of course, the aggregate throughput of the whole system. In particular we could compare results with throttling, BFQ, and CFQ.

Then we could write resulting numbers on the stone, and stick to them until something proves them wrong.

What do you (or others) think about it?

Hello, Paolo.

Quoted 25 lines · Paolo Valente

In this respect, for your generic, unpredictable scenario to make sense, there must exist at least one real system that meets the requirements of such a scenario. Or, if such a real system does not yet exist, it must be possible to emulate it. If it is impossible to achieve this last goal either, then I miss the usefulness of looking for solutions for such a scenario.

That said, let's define the instance(s) of the scenario that you find most representative, and let's test BFQ on it/them. Numbers will give us the answers. For example, what about all or part of the following groups:
. one cyclically doing random I/O for some second and then sequential I/O for the next seconds
. one doing, say, quasi-sequential I/O in ON/OFF cycles
. one starting an application cyclically
. one playing back or streaming a movie

For each group, we could then measure the time needed to complete each phase of I/O in each cycle, plus the responsiveness in the group starting an application, plus the frame drop in the group streaming the movie. In addition, we can measure the bandwidth/iops enjoyed by each group, plus, of course, the aggregate throughput of the whole system. In particular we could compare results with throttling, BFQ, and CFQ.

Quoted 4 lines

Then we could write resulting numbers on the stone, and stick to them until something proves them wrong.

What do you (or others) think about it?

That sounds great and yeah it's lame that we didn't start with that. Shaohua, would it be difficult to compare how bfq performs against blk-throttle?

Quoted 28 lines · Tejun Heo

Hello, Paolo.

Quoted 25 lines · Paolo Valente

In this respect, for your generic, unpredictable scenario to make sense, there must exist at least one real system that meets the requirements of such a scenario. Or, if such a real system does not yet exist, it must be possible to emulate it. If it is impossible to achieve this last goal either, then I miss the usefulness of looking for solutions for such a scenario.

That said, let's define the instance(s) of the scenario that you find most representative, and let's test BFQ on it/them. Numbers will give us the answers. For example, what about all or part of the following groups:
. one cyclically doing random I/O for some second and then sequential I/O for the next seconds
. one doing, say, quasi-sequential I/O in ON/OFF cycles
. one starting an application cyclically
. one playing back or streaming a movie

For each group, we could then measure the time needed to complete each phase of I/O in each cycle, plus the responsiveness in the group starting an application, plus the frame drop in the group streaming the movie. In addition, we can measure the bandwidth/iops enjoyed by each group, plus, of course, the aggregate throughput of the whole system. In particular we could compare results with throttling, BFQ, and CFQ.

Quoted 8 lines

Quoted 4 lines

Then we could write resulting numbers on the stone, and stick to them until something proves them wrong.

What do you (or others) think about it?

That sounds great and yeah it's lame that we didn't start with that. Shaohua, would it be difficult to compare how bfq performs against blk-throttle?

I had a test of BFQ. I'm using BFQ found at
http://algogroup.unimore.it/people/paolo/disk_sched/sources.php. version is 4.7.0-v8r3. It's a LSI SSD, queue depth 32. I use default setting. fio script is:

[global]
ioengine=libaio
direct=1
readwrite=randread
bs=4k
runtime=60
time_based=1
file_service_type=random:36
overwrite=1
thread=0
group_reporting=1
filename=/dev/sdb
iodepth=1
numjobs=8

[groupA]
prio=2

[groupB]
new_group
prio=6

I'll change iodepth, numjobs and prio in different tests. result unit is MB/s.

iodepth=1 numjobs=1 prio 4:4
CFQ: 28:28 BFQ: 21:21 deadline: 29:29

iodepth=8 numjobs=1 prio 4:4
CFQ: 162:162 BFQ: 102:98 deadline: 205:205

iodepth=1 numjobs=8 prio 4:4
CFQ: 157:157 BFQ: 81:92 deadline: 196:197

iodepth=1 numjobs=1 prio 2:6
CFQ: 26.7:27.6 BFQ: 20:6 deadline: 29:29

iodepth=8 numjobs=1 prio 2:6
CFQ: 166:174 BFQ: 139:72 deadline: 202:202

iodepth=1 numjobs=8 prio 2:6
CFQ: 148:150 BFQ: 90:77 deadline: 198:197

Il giorno 05 ott 2016, alle ore 21:08, Shaohua Li <[email protected]> ha scritto:

Quoted 76 lines · Shaohua Li

Quoted 28 lines · Tejun Heo

Hello, Paolo.

Quoted 25 lines · Paolo Valente

In this respect, for your generic, unpredictable scenario to make sense, there must exist at least one real system that meets the requirements of such a scenario. Or, if such a real system does not yet exist, it must be possible to emulate it. If it is impossible to achieve this last goal either, then I miss the usefulness of looking for solutions for such a scenario.

That said, let's define the instance(s) of the scenario that you find most representative, and let's test BFQ on it/them. Numbers will give us the answers. For example, what about all or part of the following groups:
. one cyclically doing random I/O for some second and then sequential I/O for the next seconds
. one doing, say, quasi-sequential I/O in ON/OFF cycles
. one starting an application cyclically
. one playing back or streaming a movie

For each group, we could then measure the time needed to complete each phase of I/O in each cycle, plus the responsiveness in the group starting an application, plus the frame drop in the group streaming the movie. In addition, we can measure the bandwidth/iops enjoyed by each group, plus, of course, the aggregate throughput of the whole system. In particular we could compare results with throttling, BFQ, and CFQ.

Quoted 8 lines

Quoted 4 lines

Then we could write resulting numbers on the stone, and stick to them until something proves them wrong.

What do you (or others) think about it?

That sounds great and yeah it's lame that we didn't start with that. Shaohua, would it be difficult to compare how bfq performs against blk-throttle?

I had a test of BFQ. I'm using BFQ found at
http://algogroup.unimore.it/people/paolo/disk_sched/sources.php. version is 4.7.0-v8r3. It's a LSI SSD, queue depth 32. I use default setting. fio script is:

[global]
ioengine=libaio
direct=1
readwrite=randread
bs=4k
runtime=60
time_based=1
file_service_type=random:36
overwrite=1
thread=0
group_reporting=1
filename=/dev/sdb
iodepth=1
numjobs=8

[groupA]
prio=2

[groupB]
new_group
prio=6

I'll change iodepth, numjobs and prio in different tests. result unit is MB/s.

iodepth=1 numjobs=1 prio 4:4
CFQ: 28:28 BFQ: 21:21 deadline: 29:29

iodepth=8 numjobs=1 prio 4:4
CFQ: 162:162 BFQ: 102:98 deadline: 205:205

iodepth=1 numjobs=8 prio 4:4
CFQ: 157:157 BFQ: 81:92 deadline: 196:197

Quoted 8 lines

iodepth=1 numjobs=1 prio 2:6
CFQ: 26.7:27.6 BFQ: 20:6 deadline: 29:29

iodepth=8 numjobs=1 prio 2:6
CFQ: 166:174 BFQ: 139:72 deadline: 202:202

iodepth=1 numjobs=8 prio 2:6
CFQ: 148:150 BFQ: 90:77 deadline: 198:197

More tests:

iodepth=8 numjobs=1 prio 2:6, group A has 50M/s limit CFQ:51:207 BFQ: 51:45 deadline: 51:216

iodepth=1 numjobs=1 prio 2:6, group A bs=4k, group B bs=64k CFQ:25:249 BFQ: 23:42 deadline: 26:251

Thanks,
Shaohua
--
To unsubscribe from this list: send the line "unsubscribe linux-block" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Quoted 89 lines

Il giorno 05 ott 2016, alle ore 21:08, Shaohua Li <[email protected]> ha scritto:

Quoted 76 lines · Shaohua Li

Quoted 28 lines · Tejun Heo

Hello, Paolo.

Quoted 25 lines · Paolo Valente

In this respect, for your generic, unpredictable scenario to make sense, there must exist at least one real system that meets the requirements of such a scenario. Or, if such a real system does not yet exist, it must be possible to emulate it. If it is impossible to achieve this last goal either, then I miss the usefulness of looking for solutions for such a scenario.

That said, let's define the instance(s) of the scenario that you find most representative, and let's test BFQ on it/them. Numbers will give us the answers. For example, what about all or part of the following groups:
. one cyclically doing random I/O for some second and then sequential I/O for the next seconds
. one doing, say, quasi-sequential I/O in ON/OFF cycles
. one starting an application cyclically
. one playing back or streaming a movie

For each group, we could then measure the time needed to complete each phase of I/O in each cycle, plus the responsiveness in the group starting an application, plus the frame drop in the group streaming the movie. In addition, we can measure the bandwidth/iops enjoyed by each group, plus, of course, the aggregate throughput of the whole system. In particular we could compare results with throttling, BFQ, and CFQ.

Quoted 8 lines

Quoted 4 lines

Then we could write resulting numbers on the stone, and stick to them until something proves them wrong.

What do you (or others) think about it?

That sounds great and yeah it's lame that we didn't start with that. Shaohua, would it be difficult to compare how bfq performs against blk-throttle?

I had a test of BFQ. I'm using BFQ found at
https://urldefense.proofpoint.com/v2/url?u=http-3A__algogroup.unimore.it_people_paolo_disk-5Fsched_sources.php&d=DQIFAg&c=5VD0RTtNlTh3ycd41b3MUw&r=X13hAPkxmvBro1Ug8vcKHw&m=zB09S7v2QifXXTa6f2_r6YLjiXq3AwAi7sqO4o2UfBQ&s=oMKpjQMXfWmMwHmANB-Qnrm2EdERzz9Oef7jcLkbyFg&e= . version is 4.7.0-v8r3. It's a LSI SSD, queue depth 32. I use default setting. fio script
is:

[global]
ioengine=libaio
direct=1
readwrite=randread
bs=4k
runtime=60
time_based=1
file_service_type=random:36
overwrite=1
thread=0
group_reporting=1
filename=/dev/sdb
iodepth=1
numjobs=8

[groupA]
prio=2

[groupB]
new_group
prio=6

I'll change iodepth, numjobs and prio in different tests. result unit is MB/s.

iodepth=1 numjobs=1 prio 4:4
CFQ: 28:28 BFQ: 21:21 deadline: 29:29

iodepth=8 numjobs=1 prio 4:4
CFQ: 162:162 BFQ: 102:98 deadline: 205:205

iodepth=1 numjobs=8 prio 4:4
CFQ: 157:157 BFQ: 81:92 deadline: 196:197

Quoted 8 lines

iodepth=1 numjobs=1 prio 2:6
CFQ: 26.7:27.6 BFQ: 20:6 deadline: 29:29

iodepth=8 numjobs=1 prio 2:6
CFQ: 166:174 BFQ: 139:72 deadline: 202:202

iodepth=1 numjobs=8 prio 2:6
CFQ: 148:150 BFQ: 90:77 deadline: 198:197

Quoted 8 lines

More tests:

iodepth=8 numjobs=1 prio 2:6, group A has 50M/s limit CFQ:51:207 BFQ: 51:45 deadline: 51:216

iodepth=1 numjobs=1 prio 2:6, group A bs=4k, group B bs=64k CFQ:25:249 BFQ: 23:42 deadline: 26:251

A true proportional share scheduler like BFQ works under the assumption to be the only limiter of the bandwidth of its clients. And the availability of such a scheduler should apparently make bandwidth limiting useless: once you have a mechanism that allows you to give each group the desired fraction of the bandwidth, and to redistribute excess bandwidth seamlessly when needed, what do you need additional limiting for?

But I'm not expert of any possible system configuration or requirement. So, if you have practical examples, I would really appreciate them. And I don't think it will be difficult to see what goes wrong in BFQ with external bw limitation, and to fix the problem.

Il giorno 05 ott 2016, alle ore 22:36, Shaohua Li <[email protected]> ha scritto:

Quoted 106 lines · Paolo Valente

Quoted 89 lines

Il giorno 05 ott 2016, alle ore 21:08, Shaohua Li <[email protected]> ha scritto:

Quoted 76 lines · Shaohua Li

Quoted 28 lines · Tejun Heo

Hello, Paolo.

Quoted 25 lines · Paolo Valente

In this respect, for your generic, unpredictable scenario to make sense, there must exist at least one real system that meets the requirements of such a scenario. Or, if such a real system does not yet exist, it must be possible to emulate it. If it is impossible to achieve this last goal either, then I miss the usefulness of looking for solutions for such a scenario.

That said, let's define the instance(s) of the scenario that you find most representative, and let's test BFQ on it/them. Numbers will give us the answers. For example, what about all or part of the following groups:
. one cyclically doing random I/O for some second and then sequential I/O for the next seconds
. one doing, say, quasi-sequential I/O in ON/OFF cycles
. one starting an application cyclically
. one playing back or streaming a movie

For each group, we could then measure the time needed to complete each phase of I/O in each cycle, plus the responsiveness in the group starting an application, plus the frame drop in the group streaming the movie. In addition, we can measure the bandwidth/iops enjoyed by each group, plus, of course, the aggregate throughput of the whole system. In particular we could compare results with throttling, BFQ, and CFQ.

Quoted 8 lines

Quoted 4 lines

Then we could write resulting numbers on the stone, and stick to them until something proves them wrong.

What do you (or others) think about it?

That sounds great and yeah it's lame that we didn't start with that. Shaohua, would it be difficult to compare how bfq performs against blk-throttle?

I had a test of BFQ. I'm using BFQ found at
https://urldefense.proofpoint.com/v2/url?u=http-3A__algogroup.unimore.it_people_paolo_disk-5Fsched_sources.php&d=DQIFAg&c=5VD0RTtNlTh3ycd41b3MUw&r=X13hAPkxmvBro1Ug8vcKHw&m=zB09S7v2QifXXTa6f2_r6YLjiXq3AwAi7sqO4o2UfBQ&s=oMKpjQMXfWmMwHmANB-Qnrm2EdERzz9Oef7jcLkbyFg&e= . version is 4.7.0-v8r3. It's a LSI SSD, queue depth 32. I use default setting. fio script
is:

[global]
ioengine=libaio
direct=1
readwrite=randread
bs=4k
runtime=60
time_based=1
file_service_type=random:36
overwrite=1
thread=0
group_reporting=1
filename=/dev/sdb
iodepth=1
numjobs=8

[groupA]
prio=2

[groupB]
new_group
prio=6

I'll change iodepth, numjobs and prio in different tests. result unit is MB/s.

iodepth=1 numjobs=1 prio 4:4
CFQ: 28:28 BFQ: 21:21 deadline: 29:29

iodepth=8 numjobs=1 prio 4:4
CFQ: 162:162 BFQ: 102:98 deadline: 205:205

iodepth=1 numjobs=8 prio 4:4
CFQ: 157:157 BFQ: 81:92 deadline: 196:197

Quoted 8 lines

iodepth=1 numjobs=1 prio 2:6
CFQ: 26.7:27.6 BFQ: 20:6 deadline: 29:29

iodepth=8 numjobs=1 prio 2:6
CFQ: 166:174 BFQ: 139:72 deadline: 202:202

iodepth=1 numjobs=8 prio 2:6
CFQ: 148:150 BFQ: 90:77 deadline: 198:197

Quoted 8 lines

More tests:

iodepth=8 numjobs=1 prio 2:6, group A has 50M/s limit CFQ:51:207 BFQ: 51:45 deadline: 51:216

iodepth=1 numjobs=1 prio 2:6, group A bs=4k, group B bs=64k CFQ:25:249 BFQ: 23:42 deadline: 26:251

A true proportional share scheduler like BFQ works under the assumption to be the only limiter of the bandwidth of its clients. And the availability of such a scheduler should apparently make bandwidth limiting useless: once you have a mechanism that allows you to give each group the desired fraction of the bandwidth, and to redistribute excess bandwidth seamlessly when needed, what do you need additional limiting for?

Quoted 5 lines

But I'm not expert of any possible system configuration or requirement. So, if you have practical examples, I would really appreciate them. And I don't think it will be difficult to see what goes wrong in BFQ with external bw limitation, and to fix the problem.

I think the test emulates a very common configuration. We assign more IO resources to high priority workload. But such workload doesn't always dispatch enough io. That's why I set a rate limit. When this happend, we hope low priority workload uses the disk bandwidth. That's the whole point of disk sharing.

Thanks,
Shaohua
--
To unsubscribe from this list: send the line "unsubscribe linux-block" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Il giorno 05 ott 2016, alle ore 20:30, Shaohua Li <[email protected]> ha scritto:

Quoted 31 lines · Tejun Heo

Hello, Paolo.

Quoted 28 lines · Paolo Valente

In this respect, for your generic, unpredictable scenario to make sense, there must exist at least one real system that meets the requirements of such a scenario. Or, if such a real system does not yet exist, it must be possible to emulate it. If it is impossible to achieve this last goal either, then I miss the usefulness of looking for solutions for such a scenario.

That said, let's define the instance(s) of the scenario that you find most representative, and let's test BFQ on it/them. Numbers will give us the answers. For example, what about all or part of the following groups:
. one cyclically doing random I/O for some second and then sequential I/O for the next seconds
. one doing, say, quasi-sequential I/O in ON/OFF cycles
. one starting an application cyclically
. one playing back or streaming a movie

For each group, we could then measure the time needed to complete each phase of I/O in each cycle, plus the responsiveness in the group starting an application, plus the frame drop in the group streaming the movie. In addition, we can measure the bandwidth/iops enjoyed by each group, plus, of course, the aggregate throughput of the whole system. In particular we could compare results with throttling, BFQ, and CFQ.

Then we could write resulting numbers on the stone, and stick to them until something proves them wrong.

Quoted 5 lines

Quoted 1 line

What do you (or others) think about it?

That sounds great and yeah it's lame that we didn't start with that. Shaohua, would it be difficult to compare how bfq performs against blk-throttle?

I had a test of BFQ.

I'm using BFQ found at
http://algogroup.unimore.it/people/paolo/disk_sched/sources.php. version is 4.7.0-v8r3.

It's a LSI SSD, queue depth 32. I use default setting. fio script is:

[global]
ioengine=libaio
direct=1
readwrite=randread
bs=4k
runtime=60
time_based=1
file_service_type=random:36
overwrite=1
thread=0
group_reporting=1
filename=/dev/sdb
iodepth=1
numjobs=8

[groupA]
prio=2

[groupB]
new_group
prio=6

I'll change iodepth, numjobs and prio in different tests. result unit is MB/s.

iodepth=1 numjobs=1 prio 4:4
CFQ: 28:28 BFQ: 21:21 deadline: 29:29

iodepth=8 numjobs=1 prio 4:4
CFQ: 162:162 BFQ: 102:98 deadline: 205:205

iodepth=1 numjobs=8 prio 4:4
CFQ: 157:157 BFQ: 81:92 deadline: 196:197

iodepth=1 numjobs=1 prio 2:6
CFQ: 26.7:27.6 BFQ: 20:6 deadline: 29:29

iodepth=8 numjobs=1 prio 2:6
CFQ: 166:174 BFQ: 139:72 deadline: 202:202

iodepth=1 numjobs=8 prio 2:6
CFQ: 148:150 BFQ: 90:77 deadline: 198:197

CFQ isn't fair at all. BFQ is very good in this side, but has poor throughput even prio is the default value.

Thanks,
Shaohua
--
To unsubscribe from this list: send the line "unsubscribe linux-block" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Il giorno 05 ott 2016, alle ore 21:47, Paolo Valente <[email protected]> ha scritto:

Quoted 35 lines

Il giorno 05 ott 2016, alle ore 20:30, Shaohua Li <[email protected]> ha scritto:

Quoted 31 lines · Tejun Heo

Hello, Paolo.

Quoted 28 lines · Paolo Valente

In this respect, for your generic, unpredictable scenario to make sense, there must exist at least one real system that meets the requirements of such a scenario. Or, if such a real system does not yet exist, it must be possible to emulate it. If it is impossible to achieve this last goal either, then I miss the usefulness of looking for solutions for such a scenario.

That said, let's define the instance(s) of the scenario that you find most representative, and let's test BFQ on it/them. Numbers will give us the answers. For example, what about all or part of the following groups:
. one cyclically doing random I/O for some second and then sequential I/O for the next seconds
. one doing, say, quasi-sequential I/O in ON/OFF cycles
. one starting an application cyclically
. one playing back or streaming a movie

For each group, we could then measure the time needed to complete each phase of I/O in each cycle, plus the responsiveness in the group starting an application, plus the frame drop in the group streaming the movie. In addition, we can measure the bandwidth/iops enjoyed by each group, plus, of course, the aggregate throughput of the whole system. In particular we could compare results with throttling, BFQ, and CFQ.

Then we could write resulting numbers on the stone, and stick to them until something proves them wrong.

Quoted 7 lines

Quoted 5 lines

Quoted 1 line

What do you (or others) think about it?

That sounds great and yeah it's lame that we didn't start with that. Shaohua, would it be difficult to compare how bfq performs against blk-throttle?

I had a test of BFQ.

Thank you very much for testing BFQ!

Quoted 3 lines

I'm using BFQ found at
http://algogroup.unimore.it/people/paolo/disk_sched/sources.php. version is 4.7.0-v8r3.

That's the latest stable version. The development version [1] already contains further improvements for fairness, latency and throughput. It is however still a release candidate.

[1] https://github.com/linusw/linux-bfq/tree/bfq-v8

Quoted 39 lines

It's a LSI SSD, queue depth 32. I use default setting. fio script is:

[global]
ioengine=libaio
direct=1
readwrite=randread
bs=4k
runtime=60
time_based=1
file_service_type=random:36
overwrite=1
thread=0
group_reporting=1
filename=/dev/sdb
iodepth=1
numjobs=8

[groupA]
prio=2

[groupB]
new_group
prio=6

I'll change iodepth, numjobs and prio in different tests. result unit is MB/s.

iodepth=1 numjobs=1 prio 4:4
CFQ: 28:28 BFQ: 21:21 deadline: 29:29

iodepth=8 numjobs=1 prio 4:4
CFQ: 162:162 BFQ: 102:98 deadline: 205:205

iodepth=1 numjobs=8 prio 4:4
CFQ: 157:157 BFQ: 81:92 deadline: 196:197

iodepth=1 numjobs=1 prio 2:6
CFQ: 26.7:27.6 BFQ: 20:6 deadline: 29:29

Quoted 9 lines

iodepth=8 numjobs=1 prio 2:6
CFQ: 166:174 BFQ: 139:72 deadline: 202:202

iodepth=1 numjobs=8 prio 2:6
CFQ: 148:150 BFQ: 90:77 deadline: 198:197

CFQ isn't fair at all. BFQ is very good in this side, but has poor throughput even prio is the default value.

Throughput is lower with BFQ for two reasons.

First, you certainly left the low_latency in its default state, i.e., on. As explained, e.g., here [2], low_latency mode is totally geared towards maximum responsiveness and minimum latency for soft real-time applications (e.g., video players). To achieve this goal, BFQ is willing to perform more idling, when necessary. This lowers throughput (I'll get back on this at the end of the discussion of the second reason).

The second, most important reason, is that a minimum of idling is the *only* way to achieve differentiated bandwidth distribution, as you requested by setting different ioprios. I stress that this constraint is not a technological accident, but a intrinsic, logical necessity. The proof is simple, and if the following explanation is too boring or confusing, I can show it to you with any trace of sync I/O.

First, to provide differentiated service, you need per-process scheduling, i.e., schedulers in which there is a separate queue associated with each process. Now, let A be the process with higher weight (ioprio), and B the process with lower weight. Both processes are sync, thus, by definition, they issue requests as follows: a few requests (probably two, or a little bit more with larger iodepth), then a little break to wait for request completion, then the next small batch and so on. For each process, the queue associated with the process (in the scheduler) is necessarily empty on the break. As a consequence, if there is no idling, then every time A reaches its break, the scheduler has only the option to switch to B (which is extremely likely to have pending requests).

The service pattern of the processes then unavoidably becomes:

A B A B A B ...

where each letter represents a full small batch served for the process. That is, 50% of the bw for each process, and complete loss of control on the desired bandwidth distribution.

So, to sum up, the reason why BFQ achieves a lower total bw is that it behaves in the only correct way to respect weights with sync I/O, i.e., it performs a little idling. If low_latency is on, then BFQ increases idling further, and this may be have caused further bw loss in your test (but this varies greatly with devices, so you can discover it only by trying).

The bottom line is that if you do want to achieve differentiation with sync I/O, you have to pay a price in terms of bw, because of idling. Actually, the recent preemption mechanism that I have introduced in BFQ is proving so effective in preserving differentiation, that I'm tempted to try some almost idleness solution. A little of accuracy should however be sacrificed. Anyway, this is still work in progress.

Thank you very much,
Paolo

[2] http://algogroup.unimore.it/people/paolo/disk_sched/description.php

Quoted 6 lines

Thanks,
Shaohua
--
To unsubscribe from this list: send the line "unsubscribe linux-block" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
Paolo Valente
Algogroup
Dipartimento di Scienze Fisiche, Informatiche e Matematiche Via Campi 213/B
41125 Modena - Italy
http://algogroup.unimore.it/people/paolo/

--
To unsubscribe from this list: send the line "unsubscribe linux-block" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Quoted 35 lines

Il giorno 05 ott 2016, alle ore 20:30, Shaohua Li <[email protected]> ha scritto:

Quoted 31 lines · Tejun Heo

Hello, Paolo.

Quoted 28 lines · Paolo Valente

In this respect, for your generic, unpredictable scenario to make sense, there must exist at least one real system that meets the requirements of such a scenario. Or, if such a real system does not yet exist, it must be possible to emulate it. If it is impossible to achieve this last goal either, then I miss the usefulness of looking for solutions for such a scenario.

That said, let's define the instance(s) of the scenario that you find most representative, and let's test BFQ on it/them. Numbers will give us the answers. For example, what about all or part of the following groups:
. one cyclically doing random I/O for some second and then sequential I/O for the next seconds
. one doing, say, quasi-sequential I/O in ON/OFF cycles
. one starting an application cyclically
. one playing back or streaming a movie

For each group, we could then measure the time needed to complete each phase of I/O in each cycle, plus the responsiveness in the group starting an application, plus the frame drop in the group streaming the movie. In addition, we can measure the bandwidth/iops enjoyed by each group, plus, of course, the aggregate throughput of the whole system. In particular we could compare results with throttling, BFQ, and CFQ.

Then we could write resulting numbers on the stone, and stick to them until something proves them wrong.

Quoted 7 lines

Quoted 5 lines

Quoted 1 line

What do you (or others) think about it?

That sounds great and yeah it's lame that we didn't start with that. Shaohua, would it be difficult to compare how bfq performs against blk-throttle?

I had a test of BFQ.

Thank you very much for testing BFQ!

Quoted 3 lines

I'm using BFQ found at
https://urldefense.proofpoint.com/v2/url?u=http-3A__algogroup.unimore.it_people_paolo_disk-5Fsched_sources.php&d=DQIFAg&c=5VD0RTtNlTh3ycd41b3MUw&r=i6WobKxbeG3slzHSIOxTVtYIJw7qjCE6S0spDTKL-J4&m=2pG8KEx5tRymExa_K0ddKH_YvhH3qvJxELBd1_lw0-w&s=FZKEAOu2sw95y9jZio2k012cQWoLzlBWDl0NiGPVW78&e= . version is 4.7.0-v8r3.

That's the latest stable version. The development version [1] already contains further improvements for fairness, latency and throughput. It is however still a release candidate.

[1] https://github.com/linusw/linux-bfq/tree/bfq-v8

Quoted 39 lines

It's a LSI SSD, queue depth 32. I use default setting. fio script is:

[global]
ioengine=libaio
direct=1
readwrite=randread
bs=4k
runtime=60
time_based=1
file_service_type=random:36
overwrite=1
thread=0
group_reporting=1
filename=/dev/sdb
iodepth=1
numjobs=8

[groupA]
prio=2

[groupB]
new_group
prio=6

I'll change iodepth, numjobs and prio in different tests. result unit is MB/s.

iodepth=1 numjobs=1 prio 4:4
CFQ: 28:28 BFQ: 21:21 deadline: 29:29

iodepth=8 numjobs=1 prio 4:4
CFQ: 162:162 BFQ: 102:98 deadline: 205:205

iodepth=1 numjobs=8 prio 4:4
CFQ: 157:157 BFQ: 81:92 deadline: 196:197

iodepth=1 numjobs=1 prio 2:6
CFQ: 26.7:27.6 BFQ: 20:6 deadline: 29:29

Quoted 9 lines

iodepth=8 numjobs=1 prio 2:6
CFQ: 166:174 BFQ: 139:72 deadline: 202:202

iodepth=1 numjobs=8 prio 2:6
CFQ: 148:150 BFQ: 90:77 deadline: 198:197

CFQ isn't fair at all. BFQ is very good in this side, but has poor throughput even prio is the default value.

Throughput is lower with BFQ for two reasons.

First, you certainly left the low_latency in its default state, i.e., on. As explained, e.g., here [2], low_latency mode is totally geared towards maximum responsiveness and minimum latency for soft real-time applications (e.g., video players). To achieve this goal, BFQ is willing to perform more idling, when necessary. This lowers throughput (I'll get back on this at the end of the discussion of the second reason).

The second, most important reason, is that a minimum of idling is the *only* way to achieve differentiated bandwidth distribution, as you requested by setting different ioprios. I stress that this constraint is not a technological accident, but a intrinsic, logical necessity. The proof is simple, and if the following explanation is too boring or confusing, I can show it to you with any trace of sync I/O.

First, to provide differentiated service, you need per-process scheduling, i.e., schedulers in which there is a separate queue associated with each process. Now, let A be the process with higher weight (ioprio), and B the process with lower weight. Both processes are sync, thus, by definition, they issue requests as follows: a few requests (probably two, or a little bit more with larger iodepth), then a little break to wait for request completion, then the next small batch and so on. For each process, the queue associated with the process (in the scheduler) is necessarily empty on the break. As a consequence, if there is no idling, then every time A reaches its break, the scheduler has only the option to switch to B (which is extremely likely to have pending requests).

The service pattern of the processes then unavoidably becomes:

A B A B A B ...

where each letter represents a full small batch served for the process. That is, 50% of the bw for each process, and complete loss of control on the desired bandwidth distribution.

So, to sum up, the reason why BFQ achieves a lower total bw is that it behaves in the only correct way to respect weights with sync I/O, i.e., it performs a little idling. If low_latency is on, then BFQ increases idling further, and this may be have caused further bw loss in your test (but this varies greatly with devices, so you can discover it only by trying).

The bottom line is that if you do want to achieve differentiation with sync I/O, you have to pay a price in terms of bw, because of idling. Actually, the recent preemption mechanism that I have introduced in BFQ is proving so effective in preserving differentiation, that I'm tempted to try some almost idleness solution. A little of accuracy should however be sacrificed. Anyway, this is still work in progress.

Il giorno 05 ott 2016, alle ore 22:46, Shaohua Li <[email protected]> ha scritto:

Quoted 106 lines · Paolo Valente

Quoted 35 lines

Il giorno 05 ott 2016, alle ore 20:30, Shaohua Li <[email protected]> ha scritto:

Quoted 31 lines · Tejun Heo

Hello, Paolo.

Quoted 28 lines · Paolo Valente

In this respect, for your generic, unpredictable scenario to make sense, there must exist at least one real system that meets the requirements of such a scenario. Or, if such a real system does not yet exist, it must be possible to emulate it. If it is impossible to achieve this last goal either, then I miss the usefulness of looking for solutions for such a scenario.

That said, let's define the instance(s) of the scenario that you find most representative, and let's test BFQ on it/them. Numbers will give us the answers. For example, what about all or part of the following groups:
. one cyclically doing random I/O for some second and then sequential I/O for the next seconds
. one doing, say, quasi-sequential I/O in ON/OFF cycles
. one starting an application cyclically
. one playing back or streaming a movie

For each group, we could then measure the time needed to complete each phase of I/O in each cycle, plus the responsiveness in the group starting an application, plus the frame drop in the group streaming the movie. In addition, we can measure the bandwidth/iops enjoyed by each group, plus, of course, the aggregate throughput of the whole system. In particular we could compare results with throttling, BFQ, and CFQ.

Then we could write resulting numbers on the stone, and stick to them until something proves them wrong.

Quoted 7 lines

Quoted 5 lines

Quoted 1 line

What do you (or others) think about it?

That sounds great and yeah it's lame that we didn't start with that. Shaohua, would it be difficult to compare how bfq performs against blk-throttle?

I had a test of BFQ.

Thank you very much for testing BFQ!

Quoted 3 lines

I'm using BFQ found at
https://urldefense.proofpoint.com/v2/url?u=http-3A__algogroup.unimore.it_people_paolo_disk-5Fsched_sources.php&d=DQIFAg&c=5VD0RTtNlTh3ycd41b3MUw&r=i6WobKxbeG3slzHSIOxTVtYIJw7qjCE6S0spDTKL-J4&m=2pG8KEx5tRymExa_K0ddKH_YvhH3qvJxELBd1_lw0-w&s=FZKEAOu2sw95y9jZio2k012cQWoLzlBWDl0NiGPVW78&e= . version is 4.7.0-v8r3.

That's the latest stable version. The development version [1] already contains further improvements for fairness, latency and throughput. It is however still a release candidate.

[1] https://github.com/linusw/linux-bfq/tree/bfq-v8

Quoted 39 lines

It's a LSI SSD, queue depth 32. I use default setting. fio script is:

[global]
ioengine=libaio
direct=1
readwrite=randread
bs=4k
runtime=60
time_based=1
file_service_type=random:36
overwrite=1
thread=0
group_reporting=1
filename=/dev/sdb
iodepth=1
numjobs=8

[groupA]
prio=2

[groupB]
new_group
prio=6

I'll change iodepth, numjobs and prio in different tests. result unit is MB/s.

iodepth=1 numjobs=1 prio 4:4
CFQ: 28:28 BFQ: 21:21 deadline: 29:29

iodepth=8 numjobs=1 prio 4:4
CFQ: 162:162 BFQ: 102:98 deadline: 205:205

iodepth=1 numjobs=8 prio 4:4
CFQ: 157:157 BFQ: 81:92 deadline: 196:197

iodepth=1 numjobs=1 prio 2:6
CFQ: 26.7:27.6 BFQ: 20:6 deadline: 29:29

Quoted 9 lines

iodepth=8 numjobs=1 prio 2:6
CFQ: 166:174 BFQ: 139:72 deadline: 202:202

iodepth=1 numjobs=8 prio 2:6
CFQ: 148:150 BFQ: 90:77 deadline: 198:197

CFQ isn't fair at all. BFQ is very good in this side, but has poor throughput even prio is the default value.

Throughput is lower with BFQ for two reasons.

Quoted 7 lines

First, you certainly left the low_latency in its default state, i.e., on. As explained, e.g., here [2], low_latency mode is totally geared towards maximum responsiveness and minimum latency for soft real-time applications (e.g., video players). To achieve this goal, BFQ is willing to perform more idling, when necessary. This lowers throughput (I'll get back on this at the end of the discussion of the second reason).

changing low_latency to 0 seems not change anything, at least for the test:
iodepth=1 numjobs=1 prio 2:6 A bs 4k:64k

Quoted 35 lines

The second, most important reason, is that a minimum of idling is the *only* way to achieve differentiated bandwidth distribution, as you requested by setting different ioprios. I stress that this constraint is not a technological accident, but a intrinsic, logical necessity. The proof is simple, and if the following explanation is too boring or confusing, I can show it to you with any trace of sync I/O.

First, to provide differentiated service, you need per-process scheduling, i.e., schedulers in which there is a separate queue associated with each process. Now, let A be the process with higher weight (ioprio), and B the process with lower weight. Both processes are sync, thus, by definition, they issue requests as follows: a few requests (probably two, or a little bit more with larger iodepth), then a little break to wait for request completion, then the next small batch and so on. For each process, the queue associated with the process (in the scheduler) is necessarily empty on the break. As a consequence, if there is no idling, then every time A reaches its break, the scheduler has only the option to switch to B (which is extremely likely to have pending requests).

The service pattern of the processes then unavoidably becomes:

A B A B A B ...

where each letter represents a full small batch served for the process. That is, 50% of the bw for each process, and complete loss of control on the desired bandwidth distribution.

So, to sum up, the reason why BFQ achieves a lower total bw is that it behaves in the only correct way to respect weights with sync I/O, i.e., it performs a little idling. If low_latency is on, then BFQ increases idling further, and this may be have caused further bw loss in your test (but this varies greatly with devices, so you can discover it only by trying).

Quoted 6 lines

The bottom line is that if you do want to achieve differentiation with sync I/O, you have to pay a price in terms of bw, because of idling. Actually, the recent preemption mechanism that I have introduced in BFQ is proving so effective in preserving differentiation, that I'm tempted to try some almost idleness solution. A little of accuracy should however be sacrificed. Anyway, this is still work in progress.

Yep, I fully understand why idle is required here. As long as workload io depth is lower than queue io depth, idle is the only way to maintain fairness. This is the core of CFQ, I bet the same for BFQ. Unfortunately idle disk harms throughput too much especially for high end SSD.

From the first test, you see that the total bw achievable by the

Thanks,
Shaohua
--
To unsubscribe from this list: send the line "unsubscribe linux-block" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Il giorno 06 ott 2016, alle ore 09:58, Paolo Valente <[email protected]> ha scritto:

Quoted 163 lines

Il giorno 05 ott 2016, alle ore 22:46, Shaohua Li <[email protected]> ha scritto:

Quoted 106 lines · Paolo Valente

Quoted 35 lines

Il giorno 05 ott 2016, alle ore 20:30, Shaohua Li <[email protected]> ha scritto:

Quoted 31 lines · Tejun Heo

Hello, Paolo.

Quoted 28 lines · Paolo Valente

In this respect, for your generic, unpredictable scenario to make sense, there must exist at least one real system that meets the requirements of such a scenario. Or, if such a real system does not yet exist, it must be possible to emulate it. If it is impossible to achieve this last goal either, then I miss the usefulness of looking for solutions for such a scenario.

That said, let's define the instance(s) of the scenario that you find most representative, and let's test BFQ on it/them. Numbers will give us the answers. For example, what about all or part of the following groups:
. one cyclically doing random I/O for some second and then sequential I/O for the next seconds
. one doing, say, quasi-sequential I/O in ON/OFF cycles
. one starting an application cyclically
. one playing back or streaming a movie

For each group, we could then measure the time needed to complete each phase of I/O in each cycle, plus the responsiveness in the group starting an application, plus the frame drop in the group streaming the movie. In addition, we can measure the bandwidth/iops enjoyed by each group, plus, of course, the aggregate throughput of the whole system. In particular we could compare results with throttling, BFQ, and CFQ.

Then we could write resulting numbers on the stone, and stick to them until something proves them wrong.

Quoted 7 lines

Quoted 5 lines

Quoted 1 line

What do you (or others) think about it?

That sounds great and yeah it's lame that we didn't start with that. Shaohua, would it be difficult to compare how bfq performs against blk-throttle?

I had a test of BFQ.

Thank you very much for testing BFQ!

Quoted 3 lines

I'm using BFQ found at
https://urldefense.proofpoint.com/v2/url?u=http-3A__algogroup.unimore.it_people_paolo_disk-5Fsched_sources.php&d=DQIFAg&c=5VD0RTtNlTh3ycd41b3MUw&r=i6WobKxbeG3slzHSIOxTVtYIJw7qjCE6S0spDTKL-J4&m=2pG8KEx5tRymExa_K0ddKH_YvhH3qvJxELBd1_lw0-w&s=FZKEAOu2sw95y9jZio2k012cQWoLzlBWDl0NiGPVW78&e= . version is 4.7.0-v8r3.

That's the latest stable version. The development version [1] already contains further improvements for fairness, latency and throughput. It is however still a release candidate.

[1] https://github.com/linusw/linux-bfq/tree/bfq-v8

Quoted 39 lines

It's a LSI SSD, queue depth 32. I use default setting. fio script is:

[global]
ioengine=libaio
direct=1
readwrite=randread
bs=4k
runtime=60
time_based=1
file_service_type=random:36
overwrite=1
thread=0
group_reporting=1
filename=/dev/sdb
iodepth=1
numjobs=8

[groupA]
prio=2

[groupB]
new_group
prio=6

I'll change iodepth, numjobs and prio in different tests. result unit is MB/s.

iodepth=1 numjobs=1 prio 4:4
CFQ: 28:28 BFQ: 21:21 deadline: 29:29

iodepth=8 numjobs=1 prio 4:4
CFQ: 162:162 BFQ: 102:98 deadline: 205:205

iodepth=1 numjobs=8 prio 4:4
CFQ: 157:157 BFQ: 81:92 deadline: 196:197

iodepth=1 numjobs=1 prio 2:6
CFQ: 26.7:27.6 BFQ: 20:6 deadline: 29:29

Quoted 9 lines

iodepth=8 numjobs=1 prio 2:6
CFQ: 166:174 BFQ: 139:72 deadline: 202:202

iodepth=1 numjobs=8 prio 2:6
CFQ: 148:150 BFQ: 90:77 deadline: 198:197

CFQ isn't fair at all. BFQ is very good in this side, but has poor throughput even prio is the default value.

Throughput is lower with BFQ for two reasons.

Quoted 7 lines

First, you certainly left the low_latency in its default state, i.e., on. As explained, e.g., here [2], low_latency mode is totally geared towards maximum responsiveness and minimum latency for soft real-time applications (e.g., video players). To achieve this goal, BFQ is willing to perform more idling, when necessary. This lowers throughput (I'll get back on this at the end of the discussion of the second reason).

changing low_latency to 0 seems not change anything, at least for the test:
iodepth=1 numjobs=1 prio 2:6 A bs 4k:64k

Quoted 35 lines

The second, most important reason, is that a minimum of idling is the *only* way to achieve differentiated bandwidth distribution, as you requested by setting different ioprios. I stress that this constraint is not a technological accident, but a intrinsic, logical necessity. The proof is simple, and if the following explanation is too boring or confusing, I can show it to you with any trace of sync I/O.

First, to provide differentiated service, you need per-process scheduling, i.e., schedulers in which there is a separate queue associated with each process. Now, let A be the process with higher weight (ioprio), and B the process with lower weight. Both processes are sync, thus, by definition, they issue requests as follows: a few requests (probably two, or a little bit more with larger iodepth), then a little break to wait for request completion, then the next small batch and so on. For each process, the queue associated with the process (in the scheduler) is necessarily empty on the break. As a consequence, if there is no idling, then every time A reaches its break, the scheduler has only the option to switch to B (which is extremely likely to have pending requests).

The service pattern of the processes then unavoidably becomes:

A B A B A B ...

where each letter represents a full small batch served for the process. That is, 50% of the bw for each process, and complete loss of control on the desired bandwidth distribution.

So, to sum up, the reason why BFQ achieves a lower total bw is that it behaves in the only correct way to respect weights with sync I/O, i.e., it performs a little idling. If low_latency is on, then BFQ increases idling further, and this may be have caused further bw loss in your test (but this varies greatly with devices, so you can discover it only by trying).

Quoted 6 lines

The bottom line is that if you do want to achieve differentiation with sync I/O, you have to pay a price in terms of bw, because of idling. Actually, the recent preemption mechanism that I have introduced in BFQ is proving so effective in preserving differentiation, that I'm tempted to try some almost idleness solution. A little of accuracy should however be sacrificed. Anyway, this is still work in progress.

Quoted 5 lines

Yep, I fully understand why idle is required here. As long as workload io depth is lower than queue io depth, idle is the only way to maintain fairness. This is the core of CFQ, I bet the same for BFQ. Unfortunately idle disk harms throughput too much especially for high end SSD.

Then I'm afraid I have to give you very bad news: bw limiting causes the same throughput loss. You can see it from your very tests. Here is one of your results with BFQ (one that is likely to have been affected less by the fact that you left low_latency on, or by further issues that I may have not yet addressed thoroughly):

iodepth=8 numjobs=1 prio 2:6
CFQ: 166:174 BFQ: 139:72 deadline: 202:202

Here is, instead, your test with bw limitation:
iodepth=8 numjobs=1 prio 2:6, group A has 50M/s limit CFQ:51:207 BFQ: 51:45 deadline: 51:216

From the first test, you see that the total bw achievable by the device is at least 404MB/s. But in the second test you get at most 267MB/s, with deadline. In this respect, the total bw achieved by BFQ in the first test is 211MB/s.

So, both throttling and proportional share need to waste bw, BFQ looses about 13% more of the total bw. In return, it gives you incomparably better bw and latency guarantees, while allowing you to configure your system with zero or minimal effort. In contrast, using bw limits to properly configure a common system like, e.g., a large file server, may become a nightmare for a sysadmin. For example, if, in the simplest case, she/he configures limits for the worst-case, then per-client limits will have to be extremely low. But the system is large and dynamic, so the actual number of clients and at the actual bw consumed by each client will vary without a break, even in the short-medium term. The bw redistribution heuristic do not give any provable guarantees on accuracy of bw redistribution. The result will likely be highly varying client bandwidths, with unlucky clients unjustly limited to low limits, and then experiencing high latencies. The latter will be further emphasized by the intrinsically bursty nature of throttling.

In addition, the scenario in your tests is the worst case for a proportional share solution: in a generic system, such as the the file server above, part of the workload is likely to be sequential or quasi-sequential (at least in medium-length time intervals), and this is enough to get very close to peak bw with a proportional-share scheduler. No configuration needed. With bw throttling, you must do the math very well to get peak bw all the time in a dynamic system.

To sum up, I do think that bw throttling is the best possible solution (and in particular your solution is very good) till blk-mq lacks an accurate scheduler. But, where you do have such a scheduler, it makes very little sense to give users a very bad service, for fear of wasting at most 13% of the bw in a single worst case.

Thanks,
Paolo

Quoted 6 lines

Thanks,
Shaohua
--
To unsubscribe from this list: send the line "unsubscribe linux-block" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
Paolo Valente
Algogroup
Dipartimento di Scienze Fisiche, Informatiche e Matematiche Via Campi 213/B
41125 Modena - Italy
http://algogroup.unimore.it/people/paolo/

--
To unsubscribe from this list: send the line "unsubscribe linux-block" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Shaohua, I have just realized that I have unconsciously defended a wrong argument. Although all the facts that I have reported are evidently true, I have argued as if the question was: "do we need to throw away throttling because there is proportional, or do we need to throw away proportional share because there is throttling?". This question is simply wrong, as I think consciously (sorry for my dissociated behavior :) ).

The best goal to achieve is to have both a good throttling mechanism, and a good proportional share scheduler. This goal would be valid if even if there was just one important scenario for each of the two approaches. The vulnus here is that you guys are constantly, and rightly, working on solutions to achieve and consolidate reasonable QoS guarantees, but an apparently very good proportional-share scheduler has been kept off for years. If you (or others) have good arguments to support this state of affairs, then this would probably be an important point to discuss.

Il giorno 06 ott 2016, alle ore 19:49, Vivek Goyal <[email protected]> ha scritto:

On Thu, Oct 06, 2016 at 03:15:50PM +0200, Paolo Valente wrote:

[..]

Quoted 7 lines

Shaohua, I have just realized that I have unconsciously defended a wrong argument. Although all the facts that I have reported are evidently true, I have argued as if the question was: "do we need to throw away throttling because there is proportional, or do we need to throw away proportional share because there is throttling?". This question is simply wrong, as I think consciously (sorry for my dissociated behavior :) ).

I was wondering about the same. We need both and both should be able to work with fast devices of today using blk-mq interfaces without much overhead.

Quoted 10 lines

The best goal to achieve is to have both a good throttling mechanism, and a good proportional share scheduler. This goal would be valid if even if there was just one important scenario for each of the two approaches. The vulnus here is that you guys are constantly, and rightly, working on solutions to achieve and consolidate reasonable QoS guarantees, but an apparently very good proportional-share scheduler has been kept off for years. If you (or others) have good arguments to support this state of affairs, then this would probably be an important point to discuss.

Paolo, CFQ is legacy now and if we can come up with a proportional IO mechanism which works reasonably well with fast devices using blk-mq interfaces, that will be much more interesting.

Vivek

Quoted 29 lines

Il giorno 06 ott 2016, alle ore 19:49, Vivek Goyal <[email protected]> ha scritto:

On Thu, Oct 06, 2016 at 03:15:50PM +0200, Paolo Valente wrote:

[..]

Quoted 7 lines

Shaohua, I have just realized that I have unconsciously defended a wrong argument. Although all the facts that I have reported are evidently true, I have argued as if the question was: "do we need to throw away throttling because there is proportional, or do we need to throw away proportional share because there is throttling?". This question is simply wrong, as I think consciously (sorry for my dissociated behavior :) ).

I was wondering about the same. We need both and both should be able to work with fast devices of today using blk-mq interfaces without much overhead.

Quoted 10 lines

The best goal to achieve is to have both a good throttling mechanism, and a good proportional share scheduler. This goal would be valid if even if there was just one important scenario for each of the two approaches. The vulnus here is that you guys are constantly, and rightly, working on solutions to achieve and consolidate reasonable QoS guarantees, but an apparently very good proportional-share scheduler has been kept off for years. If you (or others) have good arguments to support this state of affairs, then this would probably be an important point to discuss.

Quoted 4 lines

Paolo, CFQ is legacy now and if we can come up with a proportional IO mechanism which works reasonably well with fast devices using blk-mq interfaces, that will be much more interesting.

That's absolutely true. But, why do we pretend not to know that, for (at least) hundreds of thousands of users Linux will go on giving bad responsiveness, starvation, high latency and unfairness, until blk will not be used any more (assuming that these problems will somehow disappear will blk-mq). Many of these users are fully aware of these Linux long-standing problems. We could solve these problems by just adding a scheduler that has already been adopted, and thus extensively tested, by thousands of users. And more and more people are aware of this fact too. Are we doing the right thing?

Il giorno 06 ott 2016, alle ore 20:32, Vivek Goyal <[email protected]> ha scritto:

Quoted 34 lines · Paolo Valente

Quoted 29 lines

Il giorno 06 ott 2016, alle ore 19:49, Vivek Goyal <[email protected]> ha scritto:

On Thu, Oct 06, 2016 at 03:15:50PM +0200, Paolo Valente wrote:

[..]

Quoted 7 lines

Shaohua, I have just realized that I have unconsciously defended a wrong argument. Although all the facts that I have reported are evidently true, I have argued as if the question was: "do we need to throw away throttling because there is proportional, or do we need to throw away proportional share because there is throttling?". This question is simply wrong, as I think consciously (sorry for my dissociated behavior :) ).

I was wondering about the same. We need both and both should be able to work with fast devices of today using blk-mq interfaces without much overhead.

Quoted 10 lines

The best goal to achieve is to have both a good throttling mechanism, and a good proportional share scheduler. This goal would be valid if even if there was just one important scenario for each of the two approaches. The vulnus here is that you guys are constantly, and rightly, working on solutions to achieve and consolidate reasonable QoS guarantees, but an apparently very good proportional-share scheduler has been kept off for years. If you (or others) have good arguments to support this state of affairs, then this would probably be an important point to discuss.

Quoted 4 lines

Paolo, CFQ is legacy now and if we can come up with a proportional IO mechanism which works reasonably well with fast devices using blk-mq interfaces, that will be much more interesting.

Quoted 9 lines

That's absolutely true. But, why do we pretend not to know that, for (at least) hundreds of thousands of users Linux will go on giving bad responsiveness, starvation, high latency and unfairness, until blk will not be used any more (assuming that these problems will somehow disappear will blk-mq). Many of these users are fully aware of these Linux long-standing problems. We could solve these problems by just adding a scheduler that has already been adopted, and thus extensively tested, by thousands of users. And more and more people are aware of this fact too. Are we doing the right thing?

Hi Paolo,

People have been using CFQ for many years.

I am not sure if benefits
offered by BFQ over CFQ are significant enough to justify taking a completely new code and get rid of CFQ. Or are the benfits significant enough that one feels like putting time and effort into this and take chances wiht new code.

At this point of time replacing CFQ with something better is not a priority for me.

But if something better and stable goes upstream, I will gladly use it.

Vivek
--
To unsubscribe from this list: send the line "unsubscribe linux-block" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Paolo, CFQ is legacy now and if we can come up with a proportional IO mechanism which works reasonably well with fast devices using blk-mq interfaces, that will be much more interesting.

Quoted 163 lines

Il giorno 05 ott 2016, alle ore 22:46, Shaohua Li <[email protected]> ha scritto:

Quoted 106 lines · Paolo Valente

Quoted 35 lines

Il giorno 05 ott 2016, alle ore 20:30, Shaohua Li <[email protected]> ha scritto:

Quoted 31 lines · Tejun Heo

Hello, Paolo.

Quoted 28 lines · Paolo Valente

In this respect, for your generic, unpredictable scenario to make sense, there must exist at least one real system that meets the requirements of such a scenario. Or, if such a real system does not yet exist, it must be possible to emulate it. If it is impossible to achieve this last goal either, then I miss the usefulness of looking for solutions for such a scenario.

That said, let's define the instance(s) of the scenario that you find most representative, and let's test BFQ on it/them. Numbers will give us the answers. For example, what about all or part of the following groups:
. one cyclically doing random I/O for some second and then sequential I/O for the next seconds
. one doing, say, quasi-sequential I/O in ON/OFF cycles
. one starting an application cyclically
. one playing back or streaming a movie

For each group, we could then measure the time needed to complete each phase of I/O in each cycle, plus the responsiveness in the group starting an application, plus the frame drop in the group streaming the movie. In addition, we can measure the bandwidth/iops enjoyed by each group, plus, of course, the aggregate throughput of the whole system. In particular we could compare results with throttling, BFQ, and CFQ.

Then we could write resulting numbers on the stone, and stick to them until something proves them wrong.

Quoted 7 lines

Quoted 5 lines

Quoted 1 line

What do you (or others) think about it?

That sounds great and yeah it's lame that we didn't start with that. Shaohua, would it be difficult to compare how bfq performs against blk-throttle?

I had a test of BFQ.

Thank you very much for testing BFQ!

Quoted 3 lines

I'm using BFQ found at
https://urldefense.proofpoint.com/v2/url?u=http-3A__algogroup.unimore.it_people_paolo_disk-5Fsched_sources.php&d=DQIFAg&c=5VD0RTtNlTh3ycd41b3MUw&r=i6WobKxbeG3slzHSIOxTVtYIJw7qjCE6S0spDTKL-J4&m=2pG8KEx5tRymExa_K0ddKH_YvhH3qvJxELBd1_lw0-w&s=FZKEAOu2sw95y9jZio2k012cQWoLzlBWDl0NiGPVW78&e= . version is 4.7.0-v8r3.

That's the latest stable version. The development version [1] already contains further improvements for fairness, latency and throughput. It is however still a release candidate.

[1] https://github.com/linusw/linux-bfq/tree/bfq-v8

Quoted 39 lines

It's a LSI SSD, queue depth 32. I use default setting. fio script is:

[global]
ioengine=libaio
direct=1
readwrite=randread
bs=4k
runtime=60
time_based=1
file_service_type=random:36
overwrite=1
thread=0
group_reporting=1
filename=/dev/sdb
iodepth=1
numjobs=8

[groupA]
prio=2

[groupB]
new_group
prio=6

I'll change iodepth, numjobs and prio in different tests. result unit is MB/s.

iodepth=1 numjobs=1 prio 4:4
CFQ: 28:28 BFQ: 21:21 deadline: 29:29

iodepth=8 numjobs=1 prio 4:4
CFQ: 162:162 BFQ: 102:98 deadline: 205:205

iodepth=1 numjobs=8 prio 4:4
CFQ: 157:157 BFQ: 81:92 deadline: 196:197

iodepth=1 numjobs=1 prio 2:6
CFQ: 26.7:27.6 BFQ: 20:6 deadline: 29:29

Quoted 9 lines

iodepth=8 numjobs=1 prio 2:6
CFQ: 166:174 BFQ: 139:72 deadline: 202:202

iodepth=1 numjobs=8 prio 2:6
CFQ: 148:150 BFQ: 90:77 deadline: 198:197

CFQ isn't fair at all. BFQ is very good in this side, but has poor throughput even prio is the default value.

Throughput is lower with BFQ for two reasons.

Quoted 7 lines

First, you certainly left the low_latency in its default state, i.e., on. As explained, e.g., here [2], low_latency mode is totally geared towards maximum responsiveness and minimum latency for soft real-time applications (e.g., video players). To achieve this goal, BFQ is willing to perform more idling, when necessary. This lowers throughput (I'll get back on this at the end of the discussion of the second reason).

changing low_latency to 0 seems not change anything, at least for the test:
iodepth=1 numjobs=1 prio 2:6 A bs 4k:64k

Quoted 35 lines

The second, most important reason, is that a minimum of idling is the *only* way to achieve differentiated bandwidth distribution, as you requested by setting different ioprios. I stress that this constraint is not a technological accident, but a intrinsic, logical necessity. The proof is simple, and if the following explanation is too boring or confusing, I can show it to you with any trace of sync I/O.

First, to provide differentiated service, you need per-process scheduling, i.e., schedulers in which there is a separate queue associated with each process. Now, let A be the process with higher weight (ioprio), and B the process with lower weight. Both processes are sync, thus, by definition, they issue requests as follows: a few requests (probably two, or a little bit more with larger iodepth), then a little break to wait for request completion, then the next small batch and so on. For each process, the queue associated with the process (in the scheduler) is necessarily empty on the break. As a consequence, if there is no idling, then every time A reaches its break, the scheduler has only the option to switch to B (which is extremely likely to have pending requests).

The service pattern of the processes then unavoidably becomes:

A B A B A B ...

where each letter represents a full small batch served for the process. That is, 50% of the bw for each process, and complete loss of control on the desired bandwidth distribution.

So, to sum up, the reason why BFQ achieves a lower total bw is that it behaves in the only correct way to respect weights with sync I/O, i.e., it performs a little idling. If low_latency is on, then BFQ increases idling further, and this may be have caused further bw loss in your test (but this varies greatly with devices, so you can discover it only by trying).

Quoted 6 lines

The bottom line is that if you do want to achieve differentiation with sync I/O, you have to pay a price in terms of bw, because of idling. Actually, the recent preemption mechanism that I have introduced in BFQ is proving so effective in preserving differentiation, that I'm tempted to try some almost idleness solution. A little of accuracy should however be sacrificed. Anyway, this is still work in progress.

Quoted 5 lines

Yep, I fully understand why idle is required here. As long as workload io depth is lower than queue io depth, idle is the only way to maintain fairness. This is the core of CFQ, I bet the same for BFQ. Unfortunately idle disk harms throughput too much especially for high end SSD.

Then I'm afraid I have to give you very bad news: bw limiting causes the same throughput loss. You can see it from your very tests. Here is one of your results with BFQ (one that is likely to have been affected less by the fact that you left low_latency on, or by further issues that I may have not yet addressed thoroughly):

iodepth=8 numjobs=1 prio 2:6
CFQ: 166:174 BFQ: 139:72 deadline: 202:202

Here is, instead, your test with bw limitation:
iodepth=8 numjobs=1 prio 2:6, group A has 50M/s limit CFQ:51:207 BFQ: 51:45 deadline: 51:216

From the first test, you see that the total bw achievable by the device is at least 404MB/s. But in the second test you get at most 267MB/s, with deadline. In this respect, the total bw achieved by BFQ in the first test is 211MB/s.

So, both throttling and proportional share need to waste bw, BFQ looses about 13% more of the total bw.

In return, it gives you
incomparably better bw and latency guarantees, while allowing you to configure your system with zero or minimal effort. In contrast, using bw limits to properly configure a common system like, e.g., a large file server, may become a nightmare for a sysadmin. For example, if, in the simplest case, she/he configures limits for the worst-case, then per-client limits will have to be extremely low. But the system

is large and dynamic, so the actual number of clients and at the actual bw consumed by each client will vary without a break, even in the short-medium term. The bw redistribution heuristic do not give any provable guarantees on accuracy of bw redistribution. The result will likely be highly varying client bandwidths, with unlucky clients unjustly limited to low limits, and then experiencing high latencies. The latter will be further emphasized by the intrinsically bursty nature of throttling.

In addition, the scenario in your tests is the worst case for a proportional share solution: in a generic system, such as the the file server above, part of the workload is likely to be sequential or quasi-sequential (at least in medium-length time intervals), and this is enough to get very close to peak bw with a proportional-share scheduler. No configuration needed. With bw throttling, you must do the math very well to get peak bw all the time in a dynamic system.

Il giorno 06 ott 2016, alle ore 21:57, Shaohua Li <[email protected]> ha scritto:

Quoted 187 lines · Paolo Valente

Quoted 163 lines

Il giorno 05 ott 2016, alle ore 22:46, Shaohua Li <[email protected]> ha scritto:

Quoted 106 lines · Paolo Valente

Quoted 35 lines

Il giorno 05 ott 2016, alle ore 20:30, Shaohua Li <[email protected]> ha scritto:

Quoted 31 lines · Tejun Heo

Hello, Paolo.

Quoted 28 lines · Paolo Valente

In this respect, for your generic, unpredictable scenario to make sense, there must exist at least one real system that meets the requirements of such a scenario. Or, if such a real system does not yet exist, it must be possible to emulate it. If it is impossible to achieve this last goal either, then I miss the usefulness of looking for solutions for such a scenario.

That said, let's define the instance(s) of the scenario that you find most representative, and let's test BFQ on it/them. Numbers will give us the answers. For example, what about all or part of the following groups:
. one cyclically doing random I/O for some second and then sequential I/O for the next seconds
. one doing, say, quasi-sequential I/O in ON/OFF cycles
. one starting an application cyclically
. one playing back or streaming a movie

For each group, we could then measure the time needed to complete each phase of I/O in each cycle, plus the responsiveness in the group starting an application, plus the frame drop in the group streaming the movie. In addition, we can measure the bandwidth/iops enjoyed by each group, plus, of course, the aggregate throughput of the whole system. In particular we could compare results with throttling, BFQ, and CFQ.

Then we could write resulting numbers on the stone, and stick to them until something proves them wrong.

Quoted 7 lines

Quoted 5 lines

Quoted 1 line

What do you (or others) think about it?

That sounds great and yeah it's lame that we didn't start with that. Shaohua, would it be difficult to compare how bfq performs against blk-throttle?

I had a test of BFQ.

Thank you very much for testing BFQ!

Quoted 3 lines

I'm using BFQ found at
https://urldefense.proofpoint.com/v2/url?u=http-3A__algogroup.unimore.it_people_paolo_disk-5Fsched_sources.php&d=DQIFAg&c=5VD0RTtNlTh3ycd41b3MUw&r=i6WobKxbeG3slzHSIOxTVtYIJw7qjCE6S0spDTKL-J4&m=2pG8KEx5tRymExa_K0ddKH_YvhH3qvJxELBd1_lw0-w&s=FZKEAOu2sw95y9jZio2k012cQWoLzlBWDl0NiGPVW78&e= . version is 4.7.0-v8r3.

That's the latest stable version. The development version [1] already contains further improvements for fairness, latency and throughput. It is however still a release candidate.

[1] https://github.com/linusw/linux-bfq/tree/bfq-v8

Quoted 39 lines

It's a LSI SSD, queue depth 32. I use default setting. fio script is:

[global]
ioengine=libaio
direct=1
readwrite=randread
bs=4k
runtime=60
time_based=1
file_service_type=random:36
overwrite=1
thread=0
group_reporting=1
filename=/dev/sdb
iodepth=1
numjobs=8

[groupA]
prio=2

[groupB]
new_group
prio=6

I'll change iodepth, numjobs and prio in different tests. result unit is MB/s.

iodepth=1 numjobs=1 prio 4:4
CFQ: 28:28 BFQ: 21:21 deadline: 29:29

iodepth=8 numjobs=1 prio 4:4
CFQ: 162:162 BFQ: 102:98 deadline: 205:205

iodepth=1 numjobs=8 prio 4:4
CFQ: 157:157 BFQ: 81:92 deadline: 196:197

iodepth=1 numjobs=1 prio 2:6
CFQ: 26.7:27.6 BFQ: 20:6 deadline: 29:29

Quoted 9 lines

iodepth=8 numjobs=1 prio 2:6
CFQ: 166:174 BFQ: 139:72 deadline: 202:202

iodepth=1 numjobs=8 prio 2:6
CFQ: 148:150 BFQ: 90:77 deadline: 198:197

CFQ isn't fair at all. BFQ is very good in this side, but has poor throughput even prio is the default value.

Throughput is lower with BFQ for two reasons.

Quoted 7 lines

First, you certainly left the low_latency in its default state, i.e., on. As explained, e.g., here [2], low_latency mode is totally geared towards maximum responsiveness and minimum latency for soft real-time applications (e.g., video players). To achieve this goal, BFQ is willing to perform more idling, when necessary. This lowers throughput (I'll get back on this at the end of the discussion of the second reason).

changing low_latency to 0 seems not change anything, at least for the test:
iodepth=1 numjobs=1 prio 2:6 A bs 4k:64k

Quoted 35 lines

The second, most important reason, is that a minimum of idling is the *only* way to achieve differentiated bandwidth distribution, as you requested by setting different ioprios. I stress that this constraint is not a technological accident, but a intrinsic, logical necessity. The proof is simple, and if the following explanation is too boring or confusing, I can show it to you with any trace of sync I/O.

First, to provide differentiated service, you need per-process scheduling, i.e., schedulers in which there is a separate queue associated with each process. Now, let A be the process with higher weight (ioprio), and B the process with lower weight. Both processes are sync, thus, by definition, they issue requests as follows: a few requests (probably two, or a little bit more with larger iodepth), then a little break to wait for request completion, then the next small batch and so on. For each process, the queue associated with the process (in the scheduler) is necessarily empty on the break. As a consequence, if there is no idling, then every time A reaches its break, the scheduler has only the option to switch to B (which is extremely likely to have pending requests).

The service pattern of the processes then unavoidably becomes:

A B A B A B ...

where each letter represents a full small batch served for the process. That is, 50% of the bw for each process, and complete loss of control on the desired bandwidth distribution.

So, to sum up, the reason why BFQ achieves a lower total bw is that it behaves in the only correct way to respect weights with sync I/O, i.e., it performs a little idling. If low_latency is on, then BFQ increases idling further, and this may be have caused further bw loss in your test (but this varies greatly with devices, so you can discover it only by trying).

Quoted 6 lines

The bottom line is that if you do want to achieve differentiation with sync I/O, you have to pay a price in terms of bw, because of idling. Actually, the recent preemption mechanism that I have introduced in BFQ is proving so effective in preserving differentiation, that I'm tempted to try some almost idleness solution. A little of accuracy should however be sacrificed. Anyway, this is still work in progress.

Quoted 5 lines

Yep, I fully understand why idle is required here. As long as workload io depth is lower than queue io depth, idle is the only way to maintain fairness. This is the core of CFQ, I bet the same for BFQ. Unfortunately idle disk harms throughput too much especially for high end SSD.

Then I'm afraid I have to give you very bad news: bw limiting causes the same throughput loss. You can see it from your very tests. Here is one of your results with BFQ (one that is likely to have been affected less by the fact that you left low_latency on, or by further issues that I may have not yet addressed thoroughly):

iodepth=8 numjobs=1 prio 2:6
CFQ: 166:174 BFQ: 139:72 deadline: 202:202

Here is, instead, your test with bw limitation:
iodepth=8 numjobs=1 prio 2:6, group A has 50M/s limit CFQ:51:207 BFQ: 51:45 deadline: 51:216

From the first test, you see that the total bw achievable by the device is at least 404MB/s. But in the second test you get at most 267MB/s, with deadline. In this respect, the total bw achieved by BFQ in the first test is 211MB/s.

Quoted 2 lines

So, both throttling and proportional share need to waste bw, BFQ looses about 13% more of the total bw.

I don't think you calculate this correct. In iodepth 8 request size 4k, the workload can only dispatch 216M/s. Even group A doesn't dispatch any IO, group B can only dispatch 216M/s. So deadline doesn't waste any bw. CFQ wastes 216 - 207, while BFQ wastes 216 - 45. That's the problem.

Quoted 7 lines

In return, it gives you
incomparably better bw and latency guarantees, while allowing you to configure your system with zero or minimal effort. In contrast, using bw limits to properly configure a common system like, e.g., a large file server, may become a nightmare for a sysadmin. For example, if, in the simplest case, she/he configures limits for the worst-case, then per-client limits will have to be extremely low. But the system

Quoted 8 lines

is large and dynamic, so the actual number of clients and at the actual bw consumed by each client will vary without a break, even in the short-medium term. The bw redistribution heuristic do not give any provable guarantees on accuracy of bw redistribution. The result will likely be highly varying client bandwidths, with unlucky clients unjustly limited to low limits, and then experiencing high latencies. The latter will be further emphasized by the intrinsically bursty nature of throttling.

I don't disagree here. The bw/iops throttling is not easy to configure. It's kind of low level configuration, people need to know the workload very well to configure. If we have way to do proportional scheduling, everybody will cheer up. The goal of the tests is checking if proportional scheduling is feasible, the result isn't very optimistic so far.

No, I don't say BFQ isn't good. It is
much more fair than CFQ. I suppose it would work well for desktop workloads.

Quoted 7 lines

In addition, the scenario in your tests is the worst case for a proportional share solution: in a generic system, such as the the file server above, part of the workload is likely to be sequential or quasi-sequential (at least in medium-length time intervals), and this is enough to get very close to peak bw with a proportional-share scheduler. No configuration needed. With bw throttling, you must do the math very well to get peak bw all the time in a dynamic system.

I'm afraid this is not true. Workloads seldomly fully utilize the bandwidth of a highend SSD.

At least that's true here. My tests are actually pretty normal. I didn't do weird tests at all.

Thanks,
Shaohua
--
To unsubscribe from this list: send the line "unsubscribe linux-block" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Hello, Paolo.

Quoted 8 lines · Paolo Valente

Quoted 4 lines

Hmm... I think we already discussed this but here's a really simple case. There are three unknown workloads A, B and C and we want to give A certain best-effort guarantees (let's say around 80% of the underlying device) whether A is sharing the device with B or C.

That's the same example that you proposed me in our previous discussion. For this example I showed you, with many boring numbers, that with BFQ you get the most accurate distribution of the resource.

Yes, it is about the same example and what I understood was that "accurate distribution of the resources" holds as long as the randomness is incidental (ie. due to layout on the filesystem and so on) with the slice expiration mechanism offsetting the actually random workloads.

Quoted 1 line

If you have enough stamina, I can repeat them again. To save your

I'll go back to the thread and re-read them.

Quoted 11 lines

patience, here is a very brief summary. In a concrete use case, the unknown workloads turn into something like this: there will be a first time interval during which A happens to be, say, sequential, B happens to be, say, random and C happens to be, say, quasi-sequential. Then there will be a next time interval during which their characteristics change, and so on. It is easy (but boring, I acknowledge it) to show that, for each of these time intervals BFQ provides the best possible service in terms of fairness, bandwidth distribution, stability and so on. Why? Because of the elastic bandwidth-time scheduling of BFQ that we already discussed, and because BFQ is naturally accurate in redistributing aggregate throughput proportionally, when needed.

Yeah, that's what I remember and for workload above certain level of randomness its time consumption is mapped to bw, right?

Quoted 9 lines

Quoted 4 lines

I get that bfq can be a good compromise on most desktop workloads and behave reasonably well for some server workloads with the slice expiration mechanism but it really isn't an IO resource partitioning mechanism.

Right. My argument is that BFQ enables you to give to each client the bandwidth and low-latency guarantees you want. And this IMO is way better than partitioning a resource and then getting unavoidable unfairness and high latency.

But that statement only holds while bw is the main thing to guarantee, no? The level of isolation that we're looking for here is fairly strict adherence to sub/few-milliseconds in terms of high percentile scheduling latency while within the configured bw/iops limits, not "overall this device is being used pretty well".

Thanks.

--
tejun

How is this even a discussion when hard numbers, and trying any reproduction case easily reproduce the issues that CFQ causes. Reading this thread, and many others only grows not only my disappointment, but whenever someone launches kterm or scrot and their machine freezes, leaves a selective few individuals completely responsible for this. Help those users, help yourself, help Linux.

Il giorno 14 ott 2016, alle ore 18:40, Tejun Heo <[email protected]> ha scritto:

Hello, Kyle.

Quoted 6 lines · Kyle Sanderson

How is this even a discussion when hard numbers, and trying any reproduction case easily reproduce the issues that CFQ causes. Reading this thread, and many others only grows not only my disappointment, but whenever someone launches kterm or scrot and their machine freezes, leaves a selective few individuals completely responsible for this. Help those users, help yourself, help Linux.

So, just to be clear. I wasn't arguing against bfq replacing cfq (or anything along that line) but that proportional control, as implemented, would be too costly for many use cases and thus we need something along the line of what Shaohua is proposing.

FWIW, it looks like the only way we can implement proportional control on highspeed ssds with acceptable overhead

is somehow finding a way to
calculate the cost of each IO and throttle IOs according to that while controlling for latency as necessary. Slice scheduling with idling seems too expensive with highspeed devices with high io depth.

Thanks.

Signature · 1 line

tejun

That said, your 'thus' seems a little too strong: "bfq does not yet handle fast SSDs, thus we need something else". What about the millions of devices (and people) still within 10-20 K IOPS, and experiencing awful latencies and lack of bandwidth guarantees?

Quoted 2 lines

FWIW, it looks like the only way we can implement proportional control on highspeed ssds with acceptable overhead

Maybe not: as I wrote to Viveck in a previous reply, containing pointers to documentation, we have already achieved twenty millions of decisions per second with a prototype driving existing proportional-share packet schedulers (essentially without modifications).

Quoted 4 lines

is somehow finding a way to
calculate the cost of each IO and throttle IOs according to that while controlling for latency as necessary. Slice scheduling with idling seems too expensive with highspeed devices with high io depth.

Yes, that's absolutely true. I'm already thinking about an idleless solution. As I already wrote, I'm willing to help with scheduling in blk-mq. I hope there will be the opportunity to find some way to go at KS.

Il giorno 14 ott 2016, alle ore 20:35, Tejun Heo <[email protected]> ha scritto:

Hello, Paolo.

Quoted 4 lines · Paolo Valente

That said, your 'thus' seems a little too strong: "bfq does not yet handle fast SSDs, thus we need something else". What about the millions of devices (and people) still within 10-20 K IOPS, and experiencing awful latencies and lack of bandwidth guarantees?

I'm not objecting to any of that.

My point just is that bfq, at least
as currently implemented, is unfit for certain classes of use cases.

Quoted 8 lines

Quoted 2 lines

FWIW, it looks like the only way we can implement proportional control on highspeed ssds with acceptable overhead

Maybe not: as I wrote to Viveck in a previous reply, containing pointers to documentation, we have already achieved twenty millions of decisions per second with a prototype driving existing proportional-share packet schedulers (essentially without modifications).

And that doesn't require idling and thus doesn't severely impact utilization?

Quoted 9 lines

Quoted 4 lines

is somehow finding a way to
calculate the cost of each IO and throttle IOs according to that while controlling for latency as necessary. Slice scheduling with idling seems too expensive with highspeed devices with high io depth.

Yes, that's absolutely true. I'm already thinking about an idleless solution. As I already wrote, I'm willing to help with scheduling in blk-mq. I hope there will be the opportunity to find some way to go at KS.

It'd be great to have a proportional control mechanism whose overhead is acceptable. Unfortunately, we don't have one now and nothing seems right around the corner. (Mostly) work-conserving throttling would be fiddlier to use but is something which is useful regardless of such proportional control mechanism and can be obtained relatively easily.

I don't see why the two approaches would be mutually exclusive.

Thanks.

Signature · 5 lines

tejun
--
To unsubscribe from this list: send the line "unsubscribe linux-block" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Quoted 5 lines

Il giorno 14 ott 2016, alle ore 20:35, Tejun Heo <[email protected]> ha scritto:

Hello, Paolo.

Quoted 7 lines

Quoted 4 lines · Paolo Valente

That said, your 'thus' seems a little too strong: "bfq does not yet handle fast SSDs, thus we need something else". What about the millions of devices (and people) still within 10-20 K IOPS, and experiencing awful latencies and lack of bandwidth guarantees?

I'm not objecting to any of that.

Ok, sorry for misunderstanding. I'm just more and more confused about why a readily available, and not proven wrong solution has not yet been accepted, if everybody apparently acknowledges the problem.

Quoted 3 lines

My point just is that bfq, at least
as currently implemented, is unfit for certain classes of use cases.

Absolutely correct.

Quoted 12 lines

Quoted 8 lines

Quoted 2 lines

FWIW, it looks like the only way we can implement proportional control on highspeed ssds with acceptable overhead

Maybe not: as I wrote to Viveck in a previous reply, containing pointers to documentation, we have already achieved twenty millions of decisions per second with a prototype driving existing proportional-share packet schedulers (essentially without modifications).

And that doesn't require idling and thus doesn't severely impact utilization?

Nope. Packets are commonly assumed to be sent asynchronously. I guess that discussing the validity of this assumption is out of the scope of this thread.

Thanks,
Paolo

Quoted 18 lines

Quoted 9 lines

Quoted 4 lines

is somehow finding a way to
calculate the cost of each IO and throttle IOs according to that while controlling for latency as necessary. Slice scheduling with idling seems too expensive with highspeed devices with high io depth.

Yes, that's absolutely true. I'm already thinking about an idleless solution. As I already wrote, I'm willing to help with scheduling in blk-mq. I hope there will be the opportunity to find some way to go at KS.

It'd be great to have a proportional control mechanism whose overhead is acceptable. Unfortunately, we don't have one now and nothing seems right around the corner. (Mostly) work-conserving throttling would be fiddlier to use but is something which is useful regardless of such proportional control mechanism and can be obtained relatively easily.

I don't see why the two approaches would be mutually exclusive.

Quoted 8 lines

Thanks.

--
tejun
--
To unsubscribe from this list: send the line "unsubscribe linux-block" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
Paolo Valente
Algogroup
Dipartimento di Scienze Fisiche, Informatiche e Matematiche Via Campi 213/B
41125 Modena - Italy
http://algogroup.unimore.it/people/paolo/

I get that bfq can be a good compromise on most desktop workloads and behave reasonably well for some server workloads with the slice expiration mechanism but it really isn't an IO resource partitioning mechanism.

Quoted 4 lines · Tejun Heo

I get that bfq can be a good compromise on most desktop workloads and behave reasonably well for some server workloads with the slice expiration mechanism but it really isn't an IO resource partitioning mechanism.

Not just desktops, also Android phones.

So why not have BFQ as a separate scheduling policy upstream, alongside CFQ, deadline and noop?

We're already doing the per-usecase Kconfig thing for preemption. But maybe somebody already hates that and want to get rid of it, I don't know.

Quoted 11 lines · Linus Walleij

Quoted 4 lines · Tejun Heo

I get that bfq can be a good compromise on most desktop workloads and behave reasonably well for some server workloads with the slice expiration mechanism but it really isn't an IO resource partitioning mechanism.

Not just desktops, also Android phones.

So why not have BFQ as a separate scheduling policy upstream, alongside CFQ, deadline and noop?

Right.

Quoted 3 lines

We're already doing the per-usecase Kconfig thing for preemption. But maybe somebody already hates that and want to get rid of it, I don't know.

Hannes also suggested going back to making BFQ a separate scheduler rather than replacing CFQ earlier, pointing out that it mitigates against the risks of changing CFQ substantially at this point (which seems to be the biggest issue here).

Il giorno 06 ott 2016, alle ore 13:57, Austin S. Hemmelgarn <[email protected]> ha scritto:

Quoted 19 lines · Mark Brown

Quoted 11 lines · Linus Walleij

Quoted 4 lines · Tejun Heo

I get that bfq can be a good compromise on most desktop workloads and behave reasonably well for some server workloads with the slice expiration mechanism but it really isn't an IO resource partitioning mechanism.

Not just desktops, also Android phones.

So why not have BFQ as a separate scheduling policy upstream, alongside CFQ, deadline and noop?

Right.

Quoted 3 lines

We're already doing the per-usecase Kconfig thing for preemption. But maybe somebody already hates that and want to get rid of it, I don't know.

Quoted 5 lines

Hannes also suggested going back to making BFQ a separate scheduler rather than replacing CFQ earlier, pointing out that it mitigates against the risks of changing CFQ substantially at this point (which seems to be the biggest issue here).

ISTR that the original argument for this approach essentially amounted to: 'If it's so much better, why do we need both?'.

Such an argument is valid only if the new design is better in all respects (which there isn't sufficient information to decide in this case), or the negative aspects are worth the improvements (which is too workload specific to decide for something like this).

Quoted 28 lines

Il giorno 06 ott 2016, alle ore 13:57, Austin S. Hemmelgarn <[email protected]> ha scritto:

Quoted 24 lines · Mark Brown

Quoted 11 lines · Linus Walleij

Quoted 4 lines · Tejun Heo

I get that bfq can be a good compromise on most desktop workloads and behave reasonably well for some server workloads with the slice expiration mechanism but it really isn't an IO resource partitioning mechanism.

Not just desktops, also Android phones.

So why not have BFQ as a separate scheduling policy upstream, alongside CFQ, deadline and noop?

Right.

Quoted 3 lines

We're already doing the per-usecase Kconfig thing for preemption. But maybe somebody already hates that and want to get rid of it, I don't know.

Hannes also suggested going back to making BFQ a separate scheduler rather than replacing CFQ earlier, pointing out that it mitigates against the risks of changing CFQ substantially at this point (which seems to be the biggest issue here).

Quoted 3 lines

ISTR that the original argument for this approach essentially amounted to: 'If it's so much better, why do we need both?'.

Such an argument is valid only if the new design is better in all respects (which there isn't sufficient information to decide in this case), or the negative aspects are worth the improvements (which is too workload specific to decide for something like this).

All correct, apart from the workload-specific issue, which is not very clear to me. Over the last five years I have not found a single workload for which CFQ is better than BFQ, and none has been suggested.

Anyway, leaving aside this fact, IMO the real problem here is that we are in a catch-22: "we want BFQ to replace CFQ, but, since CFQ is legacy code, then you cannot change, and thus replace, CFQ"

Il giorno 06 ott 2016, alle ore 15:52, Austin S. Hemmelgarn <[email protected]> ha scritto:

Quoted 28 lines · Paolo Valente

Quoted 28 lines

Il giorno 06 ott 2016, alle ore 13:57, Austin S. Hemmelgarn <[email protected]> ha scritto:

Quoted 24 lines · Mark Brown

Quoted 11 lines · Linus Walleij

Quoted 4 lines · Tejun Heo

I get that bfq can be a good compromise on most desktop workloads and behave reasonably well for some server workloads with the slice expiration mechanism but it really isn't an IO resource partitioning mechanism.

Not just desktops, also Android phones.

So why not have BFQ as a separate scheduling policy upstream, alongside CFQ, deadline and noop?

Right.

Quoted 3 lines

We're already doing the per-usecase Kconfig thing for preemption. But maybe somebody already hates that and want to get rid of it, I don't know.

Hannes also suggested going back to making BFQ a separate scheduler rather than replacing CFQ earlier, pointing out that it mitigates against the risks of changing CFQ substantially at this point (which seems to be the biggest issue here).

Quoted 5 lines

Quoted 3 lines

ISTR that the original argument for this approach essentially amounted to: 'If it's so much better, why do we need both?'.

Such an argument is valid only if the new design is better in all respects (which there isn't sufficient information to decide in this case), or the negative aspects are worth the improvements (which is too workload specific to decide for something like this).

All correct, apart from the workload-specific issue, which is not very clear to me. Over the last five years I have not found a single workload for which CFQ is better than BFQ, and none has been suggested.

My point is that whether or not BFQ is better depends on the workload. You can't test for every workload, so you can't say definitively that BFQ is better for every workload.

At a minimum, there are workloads where the deadline and noop schedulers are better, but they're very domain specific workloads.

Based on the numbers from Shaohua, it looks like CFQ has better throughput than BFQ, and that will affect some workloads (for most, the improved fairness is worth the reduced throughput, but there probably are some cases where it isn't).

Quoted 2 lines

Anyway, leaving aside this fact, IMO the real problem here is that we are in a catch-22: "we want BFQ to replace CFQ, but, since CFQ is legacy code, then you cannot change, and thus replace, CFQ"

I agree that that's part of the issue, but I also don't entirely agree with the reasoning on it. Until blk-mq has proper I/O scheduling, people will continue to use CFQ, and based on the way things are going, it will be multiple months before that happens, whereas BFQ exists and is working now.

--
To unsubscribe from this list: send the line "unsubscribe linux-block" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Quoted 32 lines

Il giorno 06 ott 2016, alle ore 15:52, Austin S. Hemmelgarn <[email protected]> ha scritto:

Quoted 28 lines · Paolo Valente

Quoted 28 lines

Il giorno 06 ott 2016, alle ore 13:57, Austin S. Hemmelgarn <[email protected]> ha scritto:

Quoted 24 lines · Mark Brown

Quoted 11 lines · Linus Walleij

Quoted 4 lines · Tejun Heo

I get that bfq can be a good compromise on most desktop workloads and behave reasonably well for some server workloads with the slice expiration mechanism but it really isn't an IO resource partitioning mechanism.

Not just desktops, also Android phones.

So why not have BFQ as a separate scheduling policy upstream, alongside CFQ, deadline and noop?

Right.

Quoted 3 lines

We're already doing the per-usecase Kconfig thing for preemption. But maybe somebody already hates that and want to get rid of it, I don't know.

Hannes also suggested going back to making BFQ a separate scheduler rather than replacing CFQ earlier, pointing out that it mitigates against the risks of changing CFQ substantially at this point (which seems to be the biggest issue here).

Quoted 6 lines

Quoted 5 lines

Quoted 3 lines

ISTR that the original argument for this approach essentially amounted to: 'If it's so much better, why do we need both?'.

Such an argument is valid only if the new design is better in all respects (which there isn't sufficient information to decide in this case), or the negative aspects are worth the improvements (which is too workload specific to decide for something like this).

All correct, apart from the workload-specific issue, which is not very clear to me. Over the last five years I have not found a single workload for which CFQ is better than BFQ, and none has been suggested.

My point is that whether or not BFQ is better depends on the workload. You can't test for every workload, so you can't say definitively that BFQ is better for every workload.

Yes

Quoted 1 line

At a minimum, there are workloads where the deadline and noop schedulers are better, but they're very domain specific workloads.

Definitely

Quoted 1 line

Based on the numbers from Shaohua, it looks like CFQ has better throughput than BFQ, and that will affect some workloads (for most, the improved fairness is worth the reduced throughput, but there probably are some cases where it isn't).

Well, no fairness as deadline and noop, but with much less throughput than deadline and noop, doesn't sound much like the best scheduler for those workloads. With BFQ you have service guarantees, with noop or deadline you have maximum throughput.

Quoted 4 lines

Quoted 3 lines

Anyway, leaving aside this fact, IMO the real problem here is that we are in a catch-22: "we want BFQ to replace CFQ, but, since CFQ is legacy code, then you cannot change, and thus replace, CFQ"

I agree that that's part of the issue, but I also don't entirely agree with the reasoning on it. Until blk-mq has proper I/O scheduling, people will continue to use CFQ, and based on the way things are going, it will be multiple months before that happens, whereas BFQ exists and is working now.

Exactly!

Thanks,
Paolo

Not just desktops, also Android phones.

So why not have BFQ as a separate scheduling policy upstream, alongside CFQ, deadline and noop?

Hello, Vivek.

Quoted 7 lines · Vivek Goyal

Quoted 4 lines · Shaohua Li

Hi,

The background is we don't have an ioscheduler for blk-mq yet, so we can't prioritize processes/cgroups.

So this is an interim solution till we have ioscheduler for blk-mq?

It's a common permanent solution which applies to both !mq and mq.

Quoted 11 lines

Quoted 10 lines

This patch set tries to add basic arbitration
between cgroups with blk-throttle. It adds a new limit io.high for blk-throttle. It's only for cgroup2.

io.max is a hard limit throttling. cgroups with a max limit never dispatch more IO than their max limit. While io.high is a best effort throttling. cgroups with high limit can run above their high limit at appropriate time. Specifically, if all cgroups reach their high limit, all cgroups can run above their high limit. If any cgroup runs under its high limit, all other cgroups will run according to their high limit.

Quoted 4 lines

Hi Shaohua,

I still don't understand why we should not implement a weight based proportional IO mechanism and how this mechanism is better than proportional IO .

Oh, if we actually can implement proportional IO control, it'd be great. The problem is that we have no way of knowing IO cost for highspeed ssd devices. CFQ gets around the problem by using the walltime as the measure of resource usage and scheduling time slices, which works fine for rotating disks but horribly for highspeed ssds.

We can get some semblance of proportional control by just counting bw or iops but both break down badly as a means to measure the actual resource consumption depending on the workload. While limit based control is more tedious to configure, it doesn't misrepresent what's going on and is a lot less likely to produce surprising outcomes.

We *can* try to concoct something which tries to do proportional control for highspeed ssds but that's gonna be quite a bit of complexity and I'm not so sure it'd be justifiable given that we can't even figure out measurement of the most basic operating unit.

Quoted 4 lines

Agreed that we have issues with proportional IO and we don't have good solutions for these problems. But I can't see that how this mechanism will overcome these problems either.

It mostly defers the burden to the one who's configuring the limits and expects it to know the characteristics of the device and workloads and configure accordingly. It's quite a bit more tedious to use but should be able to cover good portion of use cases without being overly complicated. I agree that it'd be nice to have a simple proportional control but as you said can't see a good solution for it at the moment.

Quoted 7 lines

IIRC, biggest issue with proportional IO was that a low prio group might fill up the device queue with plenty of IO requests and later when high prio cgroup comes, it will still experience latencies anyway. And solution to the problem probably would be to get some awareness in device about priority of request and map weights to those priority. That way higher prio requests get prioritized.

Nah, the real problem is that we can't even decide what the proportions should be based on. The most fundamental part is missing.

Quoted 2 lines

Or run device at lower queue depth. That will improve latencies but migth reduce overall throughput.

And that we can't do this (and thus basically operate close to scheduling time slices) for highspeed ssds.

Quoted 3 lines

Or thorottle number of buffered writes (as Jens's writeback throttling) patches were doing. Buffered writes seem to be biggest culprit for increased latencies and being able to control these should help.

That's a different topic.

Quoted 5 lines

ioprio/weight based proportional IO mechanism is much more generic and much easier to configure for any kind of storage. io.high is absolute limit and makes it much harder to configure. One needs to know a lot about underlying volume/device's bandwidth (which varies a lot anyway based on workload).

Yeap, no disagreement there, but it still is a workable solution.

Quoted 6 lines

IMHO, we seem to be trying to cater to one specific use case using this mechanism. Something ioprio/weight based will be much more generic and we should explore implementing that along with building notion of ioprio in devices. When these two work together, we might be able to see good results. Just software mechanism alone might not be enough.

I don't think it's catering to specific use cases. It is a generic mechanism which demands knowledge and experimentation to configure. It's more a way for the kernel to cop out and defer figuring out device characteristics to userland. If you have a better idea, I'm all ears.

Agreed that we don't have a good basic unit to measure IO cost. I was thinking of measuring cost in terms of sectors as that's simple and gets more accurate on faster devices with almost no seek penalty. And

in fact this proposal is also providing fairness in terms of bandwitdh. One extra feature seems to be this notion of minimum bandwidth for each cgroup and until and unless all competing groups have met their minimum, other cgroups can't cross their limits.

(BTW, should we call io.high, io.minimum instead. To say, this is the
minimum bandwidth group should get before others get to cross their minimum limit till max limit).

Quoted 7 lines

It mostly defers the burden to the one who's configuring the limits and expects it to know the characteristics of the device and workloads and configure accordingly. It's quite a bit more tedious to use but should be able to cover good portion of use cases without being overly complicated. I agree that it'd be nice to have a simple proportional control but as you said can't see a good solution for it at the moment.

Ok, so idea is that if we can't provide something accurate in kernel, then expose a very low level knob, which is harder to configure but should work in some cases where users know their devices and workload very well.

Quoted 5 lines

I don't think it's catering to specific use cases. It is a generic mechanism which demands knowledge and experimentation to configure. It's more a way for the kernel to cop out and defer figuring out device characteristics to userland. If you have a better idea, I'm all ears.

I don't think I have a better idea as such. Once we had talked and you mentioned that for faster devices we should probably do some token based mechanism (which I believe would probably mean sector based IO accounting).

Il giorno 04 ott 2016, alle ore 20:50, Tejun Heo <[email protected]> ha scritto:

Hello, Vivek.

Quoted 3 lines · Vivek Goyal

Agreed that we don't have a good basic unit to measure IO cost. I was thinking of measuring cost in terms of sectors as that's simple and gets more accurate on faster devices with almost no seek penalty. And

If this were true, we could simply base everything on bandwidth; unfortunately, even highspeed ssds perform wildly differently depending on the specifics of workloads.

Quoted 4 lines

in fact this proposal is also providing fairness in terms of bandwitdh. One extra feature seems to be this notion of minimum bandwidth for each cgroup and until and unless all competing groups have met their minimum, other cgroups can't cross their limits.

Haven't read the patches yet but it should allow regulating in terms of both bandwidth and iops.

Quoted 3 lines

(BTW, should we call io.high, io.minimum instead. To say, this is the minimum bandwidth group should get before others get to cross their minimum limit till max limit).

The naming convetion is min, low, high, max but I'm not sure "min", which means hard minimum amount (whether guaranteed or best-effort), quite makes sense here.

Quoted 8 lines

Quoted 7 lines

It mostly defers the burden to the one who's configuring the limits and expects it to know the characteristics of the device and workloads and configure accordingly. It's quite a bit more tedious to use but should be able to cover good portion of use cases without being overly complicated. I agree that it'd be nice to have a simple proportional control but as you said can't see a good solution for it at the moment.

Quoted 4 lines

Ok, so idea is that if we can't provide something accurate in kernel, then expose a very low level knob, which is harder to configure but should work in some cases where users know their devices and workload very well.

Yeah, that's the basic idea for this approach. It'd be great if we eventually end up with proper proportional control but having something low level is useful anyway, so...

Quoted 10 lines

Quoted 5 lines

I don't think it's catering to specific use cases. It is a generic mechanism which demands knowledge and experimentation to configure. It's more a way for the kernel to cop out and defer figuring out device characteristics to userland. If you have a better idea, I'm all ears.

I don't think I have a better idea as such. Once we had talked and you mentioned that for faster devices we should probably do some token based mechanism (which I believe would probably mean sector based IO accounting).

That's more about the implementation strategy and doesn't affect whether we support bw, iops or combined configurations. In terms of implementation, I still think it'd be great to have something token based with per-cpu batch to lower the cpu overhead on highspeed devices but that shouldn't really affect the semantics.

Thanks.

Signature · 5 lines

tejun
--
To unsubscribe from this list: send the line "unsubscribe linux-block" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Quoted 4 lines · Shaohua Li

Hi,

The background is we don't have an ioscheduler for blk-mq yet, so we can't prioritize processes/cgroups.

So this is an interim solution till we have ioscheduler for blk-mq?

Quoted 10 lines

This patch set tries to add basic arbitration
between cgroups with blk-throttle. It adds a new limit io.high for blk-throttle. It's only for cgroup2.

io.max is a hard limit throttling. cgroups with a max limit never dispatch more IO than their max limit. While io.high is a best effort throttling. cgroups with high limit can run above their high limit at appropriate time. Specifically, if all cgroups reach their high limit, all cgroups can run above their high limit. If any cgroup runs under its high limit, all other cgroups will run according to their high limit.

Hi Shaohua,

I still don't understand why we should not implement a weight based proportional IO mechanism and how this mechanism is better than proportional IO .

Agreed that we have issues with proportional IO and we don't have good solutions for these problems. But I can't see that how this mechanism will overcome these problems either.

IIRC, biggest issue with proportional IO was that a low prio group might fill up the device queue with plenty of IO requests and later when high prio cgroup comes, it will still experience latencies anyway. And solution to the problem probably would be to get some awareness in device about priority of request and map weights to those priority. That way higher prio requests get prioritized.

Or run device at lower queue depth. That will improve latencies but migth reduce overall throughput.

Or thorottle number of buffered writes (as Jens's writeback throttling) patches were doing. Buffered writes seem to be biggest culprit for increased latencies and being able to control these should help.

ioprio/weight based proportional IO mechanism is much more generic and much easier to configure for any kind of storage. io.high is absolute limit and makes it much harder to configure. One needs to know a lot about underlying volume/device's bandwidth (which varies a lot anyway based on workload).
IMHO, we seem to be trying to cater to one specific use case using this mechanism. Something ioprio/weight based will be much more generic and we should explore implementing that along with building notion of ioprio in devices. When these two work together, we might be able to see good results. Just software mechanism alone might not be enough.