Skip to contentSkip to conversation

Throttling I/O

LSF/MM TOPIC Suresh Jayaraman 3 participants 9 messages Jan 25, 2013 – Jan 28, 2013 VFS

lore
Share & tools
lore
#0 lore raw

Hello,

I'd like to discuss again[1] the problem of throttling buffered writes and a throttle mechanism that works for all kinds of I/O.

Some background information.

During last year's LSF/MM, Fengguang discussed his proportional I/O controller patches as part of the writeback session. The limitations that were seen of his approach were a) non-handling of bursty IO submission in the flusher thread b) sharing config variables among different policies c) and that it violates layering and lacking long-term design. Tejun proposed back-pressure approach to the problem i.e. apply pressure where the problem is (block layer) and propagate upwards.

The general opinion at that time was that we needed more inputs/consensus needed on the natural, flexible, extensible "interface". The discussion thread that Vivek started[2] to collect the inputs on "interface", though resulted in good collection of inputs, not sure whether it represents inputs from all the interested parties.

At Kernel Summit last year, I learned from LWN[3] that the topic was discussed again. Tejun, apparently proposed a solution that splits up the global async CFQ queue by cgroup, so that the CFQ scheduler can easily schedule the per-cgroup sync/async queues according to the per-cgroup I/O weights. Fengguang proposed a solution by supporting the per-cgroup buffered write weights in balance_dirty_pages() and running a user-space daemon that updates the CFQ/BDP weights every second. There doesn't seem to be consensus towards either of the proposed approaches.

Looking at the possibility of prototyping Tejun's proposed idea lead to many questions (but my understanding may not be complete here as it is based only on LWN's mem-cg mini-summit coverage, so please correct me if wrong).

- Making cfq schedule the per cgroup sync/async queues according to I/O weights would mean that we'll need to use per cgroup cfqq's instead of per process? What will the impact on sync latencies if for example we have many sync only tasks in one cgroup and many async tasks in another? What if BLK_CGROUP is not configured, what would be the fallback behavior?

- Suppose if we have 100 cgroups and we are to have one cfqq per priority per cgroup, this would mean we'll be requiring 100 x 3 x 8 = 2400 cfqq's (3 classes and 8 priorities) in the worst case (as opposed to current 24 cfqqs)? This may not be as drastic as it sounds as we create cfqq's only on demand and we normally won't have tasks with every priority and every class?

I'm primarily interested in having the ability to limit/throttle buffered I/O on a multiuser system where one heavy I/O user shouldn't be impacting others and everyone should be getting their allocated share. I understand thought there are different possible use-cases and the agreed approach should be limiting any potential use-case and hence having a consensus is quite important. So, I think a discussion on the topic might help.

I would also be interested in other Network filesystem topics that have been already proposed including NFS Ganesha, readdirplus syscall etc. I have been working on Network filesystems for many years now and recently started looking into block layer side of things too.

[1] http://comments.gmane.org/gmane.linux.kernel.mm/74805 (Last year's proposal)
[2] http://www.spinics.net/lists/linux-fsdevel/msg53171.html
[3] http://lwn.net/Articles/516540/

Thanks

Signature · 1 line
Suresh Jayaraman
#1 lore raw
Quoted 22 lines · Suresh Jayaraman

Hello,

I'd like to discuss again[1] the problem of throttling buffered writes and a throttle mechanism that works for all kinds of I/O.

Some background information.

During last year's LSF/MM, Fengguang discussed his proportional I/O controller patches as part of the writeback session. The limitations that were seen of his approach were a) non-handling of bursty IO submission in the flusher thread b) sharing config variables among different policies c) and that it violates layering and lacking long-term design. Tejun proposed back-pressure approach to the problem i.e. apply pressure where the problem is (block layer) and propagate upwards.

The general opinion at that time was that we needed more inputs/consensus needed on the natural, flexible, extensible "interface". The discussion thread that Vivek started[2] to collect the inputs on "interface", though resulted in good collection of inputs, not sure whether it represents inputs from all the interested parties.

Quoted 9 lines

At Kernel Summit last year, I learned from LWN[3] that the topic was discussed again. Tejun, apparently proposed a solution that splits up the global async CFQ queue by cgroup, so that the CFQ scheduler can easily schedule the per-cgroup sync/async queues according to the per-cgroup I/O weights. Fengguang proposed a solution by supporting the per-cgroup buffered write weights in balance_dirty_pages() and running a user-space daemon that updates the CFQ/BDP weights every second. There doesn't seem to be consensus towards either of the proposed approaches.

Moving async queues in respective cgroup is easy part. Also for throttling, you don't need CFQ. So CFQ and IO throttling are little orthogonal. (I am assuming by throttling you mean upper limiting IO).

And I think tejun wanted to implement throttling at block layer and wanted vm to adjust/respond to per group IO backlog when it comes to writting to dirty data/inodes.

Once we have take care of writeback problem then comes the issue of being able to associate a dirty inode/page to a cgroup. Not sure if something has happened on that front or not. In the past it was thought to be simple that one inode belongs to one IO cgroup.

Also seriously, in CFQ, group idling performance penalty is too high and might start showing up easily on a single spindle sata disk also. Especially given the fact that people will come up with hybrid SATA drives with some caching internally. So SATA drive will not be as slow.

So proportional group scheduling of CFQ is limited to such a specific corner case of slow SATA drive. I am not sure how many people really use it.

To me, we first need to have some ideas here on how to implement low cost proportional group scheduling (either in CFQ or in other scheduler). Till then we can develop a lot of infrastructure but it usability will be very limited.

Thanks
Vivek

#2 lore raw

Hey, guys.

Quoted 8 lines · Vivek Goyal

And I think tejun wanted to implement throttling at block layer and wanted vm to adjust/respond to per group IO backlog when it comes to writting to dirty data/inodes.

Once we have take care of writeback problem then comes the issue of being able to associate a dirty inode/page to a cgroup. Not sure if something has happened on that front or not. In the past it was thought to be simple that one inode belongs to one IO cgroup.

Yeap, the above two sum it up pretty good.

Quoted 9 lines

Also seriously, in CFQ, group idling performance penalty is too high and might start showing up easily on a single spindle sata disk also. Especially given the fact that people will come up with hybrid SATA drives with some caching internally. So SATA drive will not be as slow.

So proportional group scheduling of CFQ is limited to such a specific corner case of slow SATA drive. I am not sure how many people really use it.

I don't think so. If you personal usages, sure, it's not very useful but then again proportional IO control itself isn't all that useful for personal use, but if you go to backend infrastructure requiring a lot of capacity, spindled drives still rule the roost and large deployment of on-device flash cache is not as immediate, if it ever happens, that is. Spindle drives may not be in your desk/laptops, but they continue to be deployed massively in the backend.

For example, google has been using half-hacky hierarchical writeback support in cfq for quite some time now and they'll switch to upstream implementation once we get it working, so I don't think it's a wasted effort.

Thanks.

Signature · 1 line
tejun
#3 lore raw
Quoted 24 lines · Tejun Heo

Hey, guys.

Quoted 8 lines · Vivek Goyal

And I think tejun wanted to implement throttling at block layer and wanted vm to adjust/respond to per group IO backlog when it comes to writting to dirty data/inodes.

Once we have take care of writeback problem then comes the issue of being able to associate a dirty inode/page to a cgroup. Not sure if something has happened on that front or not. In the past it was thought to be simple that one inode belongs to one IO cgroup.

Yeap, the above two sum it up pretty good.

Quoted 9 lines

Also seriously, in CFQ, group idling performance penalty is too high and might start showing up easily on a single spindle sata disk also. Especially given the fact that people will come up with hybrid SATA drives with some caching internally. So SATA drive will not be as slow.

So proportional group scheduling of CFQ is limited to such a specific corner case of slow SATA drive. I am not sure how many people really use it.

Quoted 5 lines

I don't think so. If you personal usages, sure, it's not very useful but then again proportional IO control itself isn't all that useful for personal use, but if you go to backend infrastructure requiring a lot of capacity, spindled drives still rule the roost and large deployment of on-device flash cache is not as immediate,

Hi Tejun,

How many of these spindle drives are not behind some kind of hardware raid or on SAN network. Becaue any aggregation of spindle drives by hardware/external entity makes group scheduling not worth it very soon.

Quoted 5 lines

For example, google has been using half-hacky hierarchical writeback support in cfq for quite some time now and they'll switch to upstream implementation once we get it working, so I don't think it's a wasted effort.

I guess apart from google I have not heard anybody else using it successfully and that's when I get skeptic about it. May be once the support for buffered write control is in, things will be better. Because, that's the biggest offending workload people want to protect against.

Thanks
Vivek

#4 lore raw

Hey,

Quoted 3 lines · Vivek Goyal

How many of these spindle drives are not behind some kind of hardware raid or on SAN network. Becaue any aggregation of spindle drives by hardware/external entity makes group scheduling not worth it very soon.

Hmmm? Sure, enterprise storage is usually behind some SAN silliness, but if you go larger scale, e.g. the "cloud" stuff, people usually go for the most economic solution - ie. some spindles per machine. At least the ones I know do.

Quoted 9 lines
Quoted 4 lines

For example, google has been using half-hacky hierarchical writeback support in cfq for quite some time now and they'll switch to upstream implementation once we get it working, so I don't think it's a wasted effort.

I guess apart from google I have not heard anybody else using it successfully and that's when I get skeptic about it. May be once the support for buffered write control is in, things will be better. Because, that's the biggest offending workload people want to protect against.

Yeah, I think it's mostly because we don't support it and there aren't too many organizations with enough muscle to implement and maintain their own hierarchical iosched and mutilations to writeback path. Given the workloads these servers have to run, some form of io resource control is just necessary.

Thanks.

Signature · 1 line
tejun
#5 lore raw
In reply to Vivek Goyal
Quoted 33 lines · Vivek Goyal
Quoted 22 lines · Suresh Jayaraman

Hello,

I'd like to discuss again[1] the problem of throttling buffered writes and a throttle mechanism that works for all kinds of I/O.

Some background information.

During last year's LSF/MM, Fengguang discussed his proportional I/O controller patches as part of the writeback session. The limitations that were seen of his approach were a) non-handling of bursty IO submission in the flusher thread b) sharing config variables among different policies c) and that it violates layering and lacking long-term design. Tejun proposed back-pressure approach to the problem i.e. apply pressure where the problem is (block layer) and propagate upwards.

The general opinion at that time was that we needed more inputs/consensus needed on the natural, flexible, extensible "interface". The discussion thread that Vivek started[2] to collect the inputs on "interface", though resulted in good collection of inputs, not sure whether it represents inputs from all the interested parties.

Quoted 9 lines

At Kernel Summit last year, I learned from LWN[3] that the topic was discussed again. Tejun, apparently proposed a solution that splits up the global async CFQ queue by cgroup, so that the CFQ scheduler can easily schedule the per-cgroup sync/async queues according to the per-cgroup I/O weights. Fengguang proposed a solution by supporting the per-cgroup buffered write weights in balance_dirty_pages() and running a user-space daemon that updates the CFQ/BDP weights every second. There doesn't seem to be consensus towards either of the proposed approaches.

Quoted 3 lines

Moving async queues in respective cgroup is easy part. Also for throttling, you don't need CFQ. So CFQ and IO throttling are little orthogonal. (I am assuming by throttling you mean upper limiting IO).

I meant being able to limit I/O. I didn't mean to strictly distinguish between "Upper limit" and "Proportional I/O control" in this context because my understanding is that limiting I/O for a customer on how much he is paying could be achieved using proportional control as well (by doing a little math with the weights). Perhaps, the topic name might have suggested that the discussion is only about "Upper limit" and not so much about "Proportional I/O control". But, my intent is to arrive at an acceptable mechanism that will allow limiting I/O.

Quoted 8 lines

And I think tejun wanted to implement throttling at block layer and wanted vm to adjust/respond to per group IO backlog when it comes to writting to dirty data/inodes.

Once we have take care of writeback problem then comes the issue of being able to associate a dirty inode/page to a cgroup. Not sure if something has happened on that front or not. In the past it was thought to be simple that one inode belongs to one IO cgroup.

Yes, this was discussed last year. But, not so much happened AFAIK.

Thanks

Signature · 1 line
Suresh Jayaraman
#6 lore raw

Hey, Suresh.

Quoted 10 lines · Suresh Jayaraman
Quoted 8 lines

And I think tejun wanted to implement throttling at block layer and wanted vm to adjust/respond to per group IO backlog when it comes to writting to dirty data/inodes.

Once we have take care of writeback problem then comes the issue of being able to associate a dirty inode/page to a cgroup. Not sure if something has happened on that front or not. In the past it was thought to be simple that one inode belongs to one IO cgroup.

Yes, this was discussed last year. But, not so much happened AFAIK.

Yeah, mostly because there were many more pressing issues around cgroup and blkcg. Hierarchical support for cfq is now pending and a lot of the foundation work for unified hierarchy (which IMHO is essential for sane interaction between memcg and blkcg for writeback) has been done, so it's getting there.

Thanks.

Signature · 1 line
tejun
#7 lore raw
In reply to Suresh Jayaraman

Hey, Suresh.

Quoted 6 lines · Suresh Jayaraman

- Making cfq schedule the per cgroup sync/async queues according to I/O weights would mean that we'll need to use per cgroup cfqq's instead of per process? What will the impact on sync latencies if for example we have many sync only tasks in one cgroup and many async tasks in another? What if BLK_CGROUP is not configured, what would be the fallback behavior?

So, we currently have synd cfqqs in cgroup cfqgs and shared cfqqs in the root cfqg. The end result would be splitting shared cfqqs into cgroup cfqgs. We may have to change how cfqgs are chosen depending on whether it only has async IOs pending. Not sure.

Quoted 6 lines

- Suppose if we have 100 cgroups and we are to have one cfqq per priority per cgroup, this would mean we'll be requiring 100 x 3 x 8 = 2400 cfqq's (3 classes and 8 priorities) in the worst case (as opposed to current 24 cfqqs)? This may not be as drastic as it sounds as we create cfqq's only on demand and we normally won't have tasks with every priority and every class?

I don't think that's a problem. We already have a cfqq per active IO context which can go way beyond 10k depending on work load.

Thanks.

Signature · 1 line
tejun
#8 lore raw
Quoted 10 lines · Tejun Heo

Hey, Suresh.

Quoted 6 lines · Suresh Jayaraman

- Making cfq schedule the per cgroup sync/async queues according to I/O weights would mean that we'll need to use per cgroup cfqq's instead of per process? What will the impact on sync latencies if for example we have many sync only tasks in one cgroup and many async tasks in another? What if BLK_CGROUP is not configured, what would be the fallback behavior?

Quoted 4 lines

So, we currently have synd cfqqs in cgroup cfqgs and shared cfqqs in the root cfqg. The end result would be splitting shared cfqqs into cgroup cfqgs. We may have to change how cfqgs are chosen depending on whether it only has async IOs pending. Not sure.

Ah, ok. Even if we have a way to check if in a particular cgroup all I/O is async or not, I have feeling that sync latencies might still get impacted for e.g. if we have a very few sync tasks plus many async tasks in one cgroup competing with all sync tasks in another group or some other combinations, no?

Thanks

Signature · 1 line
Suresh Jayaraman