Skip to contentSkip to conversation

IO Controller

RFC In review Vivek Goyal 31 participants 160 messages Mar 12, 2009 – May 5, 2009 42 messages not in the archive Page 1 of 2

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

Hi All,

Here is another posting for IO controller patches. Last time I had posted RFC patches for an IO controller which did bio control per cgroup.

http://lkml.org/lkml/2008/11/6/227

One of the takeaway from the discussion in this thread was that let us implement a common layer which contains the proportional weight scheduling code which can be shared by all the IO schedulers.

Implementing IO controller will not cover the devices which don't use IO schedulers but it should cover the common case.

There were more discussions regarding 2 level vs 1 level IO control at following link.

https://lists.linux-foundation.org/pipermail/containers/2009-January/015402.html

So in the mean time we took the discussion off the list and spent time on making the 1 level control apporoach work where majority of the proportional weight control is shared by the four schedulers instead of each one having to replicate the code. We make use of BFQ code for fair queuing as posted by Paolo and Fabio here.

http://lkml.org/lkml/2008/11/11/148

Details about design and howto have been put in documentation patch.

I have done very basic testing of running 2 or 3 "dd" threads in different cgroups. Wanted to get the patchset out for feedback/review before we dive into more bug fixing, benchmarking, optimizations etc.

Your feedback/comments are welcome.

Patch series contains 10 patches. It should be compilable and bootable after every patch. Intial 2 patches implement flat fair queuing (no cgroup support) and make cfq to use that. Later patches introduce hierarchical fair queuing support in elevator layer and modify other IO schdulers to use that.

Thanks
Vivek

#1 lore raw

PATCH 1/10 Documentation

o Documentation for io-controller.

---
 Documentation/block/io-controller.txt |  221 +++++++++++++++++++++++++++++++++
 1 files changed, 221 insertions(+), 0 deletions(-)
 create mode 100644 Documentation/block/io-controller.txt
Documentation/block/io-controller.txt new file +221 -0
new file mode 100644
index 0000000..8884c5a
@@ -0,0 +1,221 @@
+				IO Controller
+				=============
+
+Overview
+========
+
+This patchset implements a proportional weight IO controller. That is one
+can create cgroups and assign prio/weights to those cgroups and task group
+will get access to disk proportionate to the weight of the group.
+
+These patches modify elevator layer and individual IO schedulers to do
+IO control hence this io controller works only on block devices which use
+one of the standard io schedulers can not be used with any xyz logical block
+device.
+
+The assumption/thought behind modifying IO scheduler is that resource control
+is needed only on leaf nodes where the actual contention for resources is
+present and not on intertermediate logical block devices.
+
+Consider following hypothetical scenario. Lets say there are three physical
+disks, namely sda, sdb and sdc. Two logical volumes (lv0 and lv1) have been
+created on top of these. Some part of sdb is in lv0 and some part is in lv1.
+
+			    lv0      lv1
+			  /	\  /     \
+			sda      sdb      sdc
+
+Also consider following cgroup hierarchy
+
+				root
+				/   \
+			       A     B
+			      / \    / \
+			     T1 T2  T3  T4
+
+A and B are two cgroups and T1, T2, T3 and T4 are tasks with-in those cgroups.
+Assuming T1, T2, T3 and T4 are doing IO on lv0 and lv1. These tasks should
+get their fair share of bandwidth on disks sda, sdb and sdc. There is no
+IO control on intermediate logical block nodes (lv0, lv1).
+
+So if tasks T1 and T2 are doing IO on lv0 and T3 and T4 are doing IO on lv1
+only, there will not be any contetion for resources between group A and B if
+IO is going to sda or sdc. But if actual IO gets translated to disk sdb, then
+IO scheduler associated with the sdb will distribute disk bandwidth to
+group A and B proportionate to their weight.
+
+CFQ already has the notion of fairness and it provides differential disk
+access based on priority and class of the task. Just that it is flat and
+with cgroup stuff, it needs to be made hierarchical.
+
+Rest of the IO schedulers (noop, deadline and AS) don't have any notion
+of fairness among various threads.
+
+One of the concerns raised with modifying IO schedulers was that we don't
+want to replicate the code in all the IO schedulers. These patches share
+the fair queuing code which has been moved to a common layer (elevator
+layer). Hence we don't end up replicating code across IO schedulers.
+
+Design
+======
+This patchset primarily uses BFQ (Budget Fair Queuing) code to provide
+fairness among different IO queues. Fabio and Paolo implemented BFQ which uses
+B-WF2Q+ algorithm for fair queuing.
+
+Why BFQ?
+
+- Not sure if weighted round robin logic of CFQ can be easily extended for
+  hierarchical mode. One of the things is that we can not keep dividing
+  the time slice of parent group among childrens. Deeper we go in hierarchy
+  time slice will get smaller.
+
+  One of the ways to implement hierarchical support could be to keep track
+  of virtual time and service provided to queue/group and select a queue/group
+  for service based on any of the various available algoriths.
+
+  BFQ already had support for hierarchical scheduling, taking those patches
+  was easier.
+
+- BFQ was designed to provide tighter bounds/delay w.r.t service provided
+  to a queue. Delay/Jitter with BFQ is supposed to be O(1).
+
+  Note: BFQ originally used amount of IO done (number of sectors) as notion
+        of service provided. IOW, it tried to provide fairness in terms of
+        actual IO done and not in terms of actual time disk access was
+	given to a queue.
+
+	This patcheset modified BFQ to provide fairness in time domain because
+	that's what CFQ does. So idea was try not to deviate too much from
+	the CFQ behavior initially.
+
+	Providing fairness in time domain makes accounting trciky because
+	due to command queueing, at one time there might be multiple requests
+	from different queues and there is no easy way to find out how much
+	disk time actually was consumed by the requests of a particular
+	queue. More about this in comments in source code.
+
+So it is yet to be seen if changing to time domain still retains BFQ gurantees
+or not.
+
+From data structure point of view, one can think of a tree per device, where
+io groups and io queues are hanging and are being scheduled using B-WF2Q+
+algorithm. io_queue, is end queue where requests are actually stored and
+dispatched from (like cfqq).
+
+These io queues are primarily created by and managed by end io schedulers
+depending on its semantics. For example, noop, deadline and AS ioschedulers
+keep one io queues per cgroup and cfqq keeps one io queue per io_context in
+a cgroup (apart from async queues).
+
+A request is mapped to an io group by elevator layer and which io queue it
+is mapped to with in group depends on ioscheduler. Currently "current" task
+is used to determine the cgroup (hence io group) of the request. Down the
+line we need to make use of bio-cgroup patches to map delayed writes to
+right group.
+
+Going back to old behavior
+==========================
+In new scheme of things essentially we are creating hierarchical fair
+queuing logic in elevator layer and chaning IO schedulers to make use of
+that logic so that end IO schedulers start supporting hierarchical scheduling.
+
+Elevator layer continues to support the old interfaces. So even if fair queuing
+is enabled at elevator layer, one can have both new hierchical scheduler as
+well as old non-hierarchical scheduler operating.
+
+Also noop, deadline and AS have option of enabling hierarchical scheduling.
+If it is selected, fair queuing is done in hierarchical manner. If hierarchical
+scheduling is disabled, noop, deadline and AS should retain their existing
+behavior.
+
+CFQ is the only exception where one can not disable fair queuing as it is
+needed for provding fairness among various threads even in non-hierarchical
+mode.
+
+Various user visible config options
+===================================
+CONFIG_IOSCHED_NOOP_HIER
+	- Enables hierchical fair queuing in noop. Not selecting this option
+	  leads to old behavior of noop.
+
+CONFIG_IOSCHED_DEADLINE_HIER
+	- Enables hierchical fair queuing in deadline. Not selecting this
+	  option leads to old behavior of deadline.
+
+CONFIG_IOSCHED_AS_HIER
+	- Enables hierchical fair queuing in AS. Not selecting this option
+	  leads to old behavior of AS.
+
+CONFIG_IOSCHED_CFQ_HIER
+	- Enables hierarchical fair queuing in CFQ. Not selecting this option
+	  still does fair queuing among various queus but it is flat and not
+	  hierarchical.
+
+Config options selected automatically
+=====================================
+These config options are not user visible and are selected/deselected
+automatically based on IO scheduler configurations.
+
+CONFIG_ELV_FAIR_QUEUING
+	- Enables/Disables the fair queuing logic at elevator layer.
+
+CONFIG_GROUP_IOSCHED
+	- Enables/Disables hierarchical queuing and associated cgroup bits.
+
+TODO
+====
+- Lots of cleanups, testing, bug fixing, optimizations, benchmarking etc...
+- Convert cgroup ioprio to notion of weight.
+- Anticipatory code will need more work. It is not working properly currently
+  and needs more thought.
+- Use of bio-cgroup patches.
+- Use of Nauman's per cgroup request descriptor patches.
+
+HOWTO
+=====
+So far I have done very simple testing of running two dd threads in two
+different cgroups. Here is what you can do.
+
+- Enable hierarchical scheduling in io scheuduler of your choice (say cfq).
+	CONFIG_IOSCHED_CFQ_HIER=y
+
+- Compile and boot into kernel and mount IO controller.
+
+	mount -t cgroup -o io none /cgroup
+
+- Create two cgroups
+	mkdir -p /cgroup/test1/ /cgroup/test2
+
+- Set io priority of group test1 and test2
+	echo 0 > /cgroup/test1/io.ioprio
+	echo 4 > /cgroup/test2/io.ioprio
+
+- Create two same size files (say 512MB each) on same disk (file1, file2) and
+  launch two dd threads in different cgroup to read those files. Make sure
+  right io scheduler is being used for the block device where files are
+  present (the one you compiled in hierarchical mode).
+
+	echo 1 > /proc/sys/vm/drop_caches
+
+	dd if=/mnt/lv0/zerofile1 of=/dev/null &
+	echo $! > /cgroup/test1/tasks
+	cat /cgroup/test1/tasks
+
+	dd if=/mnt/lv0/zerofile2 of=/dev/null &
+	echo $! > /cgroup/test2/tasks
+	cat /cgroup/test2/tasks
+
+- First dd should finish first.
+
+Some Test Results
+=================
+- Two dd in two cgroups with prio 0 and 4. Ran two "dd" in those cgroups.
+
+234179072 bytes (234 MB) copied, 10.1811 s, 23.0 MB/s
+234179072 bytes (234 MB) copied, 12.6187 s, 18.6 MB/s
+
+- Three dd in three cgroups with prio 0, 4, 4.
+
+234179072 bytes (234 MB) copied, 13.7654 s, 17.0 MB/s
+234179072 bytes (234 MB) copied, 19.476 s, 12.0 MB/s
+234179072 bytes (234 MB) copied, 20.1858 s, 11.6 MB/s
Signature · 1 line
1.6.0.1
#2 lore raw

Re: [PATCH 01/10] Documentation

Documentation/block/io-controller.txt line 114

Quoted 4 lines · On Wed, 11 Mar 2009 21:56:46 -0400 Vivek Goyal…
Documentation/block/io-controller.txt new file +4 -0
+Currently "current" task
+is used to determine the cgroup (hence io group) of the request. Down the
+line we need to make use of bio-cgroup patches to map delayed writes to
+right group.

You handled this problem pretty neatly!

It's always been a BIG problem for all the io-controlling schemes, and most of them seem to have "handled" it in the above way :(

But for many workloads, writeback is the majority of the IO and it has always been the form of IO which has caused us the worst contention and latency problems. So I don't think that we can proceed with _anything_ until we at least have a convincing plan here.

Also.. there are so many IO controller implementations that I've lost track of who is doing what. I do have one private report here that Andreas's controller "is incredibly productive for us and has allowed us to put twice as many users per server with faster times for all users". Which is pretty stunning, although it should be viewed as a condemnation of the current code, I'm afraid.

So my question is: what is the definitive list of proposed-io-controller-implementations and how do I cunningly get all you guys to check each others homework? :)

A message that is not in the archive (sent off-list or not yet received): 20090312001146.74591b9d.akpm-de/[email protected]

#4 lore raw

Re: [PATCH 01/10] Documentation

Hi Andrew,

Quoted 6 lines

Also.. there are so many IO controller implementations that I've lost track of who is doing what. I do have one private report here that Andreas's controller "is incredibly productive for us and has allowed us to put twice as many users per server with faster times for all users". Which is pretty stunning, although it should be viewed as a condemnation of the current code, I'm afraid.

I'm developing dm-ioband, which is an another IO controller, and I would like to hear your comments about dm-ioband if you have.

dm-ioband web page:
http://people.valinux.co.jp/~ryov/dm-ioband/

Quoted 3 lines

So my question is: what is the definitive list of proposed-io-controller-implementations and how do I cunningly get all you guys to check each others homework? :)

Dm-ioband is implemented as a device-mapper driver, so I'm proposing dm-ioband to dm-devel and I hope device-mapper staffs take care of it.

Thanks,
Ryo Tsuruta

#5 lore raw

Re: [PATCH 01/10] Documentation

In reply to a message not in the archive
Quoted 9 lines · Andrew Morton

Documentation/block/io-controller.txt line 114

Quoted 4 lines · On Wed, 11 Mar 2009 21:56:46 -0400 Vivek Goyal…
Documentation/block/io-controller.txt new file +4 -0
+Currently "current" task
+is used to determine the cgroup (hence io group) of the request. Down the
+line we need to make use of bio-cgroup patches to map delayed writes to
+right group.

You handled this problem pretty neatly!

Quoted 8 lines

It's always been a BIG problem for all the io-controlling schemes, and most of them seem to have "handled" it in the above way :(

But for many workloads, writeback is the majority of the IO and it has always been the form of IO which has caused us the worst contention and latency problems. So I don't think that we can proceed with _anything_ until we at least have a convincing plan here.

Hi Andrew,

Nauman is already maintaining the bio-cgroup patches (originally from valinux folks) on top of this patchset for attributing write requests to correct cgroup. We did not include those in initial posting thinking that patchest will bloat further.

We can pull in bio-cgroup patches also in this series to attribute writes to right cgroup.

Quoted 8 lines

Also.. there are so many IO controller implementations that I've lost track of who is doing what. I do have one private report here that Andreas's controller "is incredibly productive for us and has allowed us to put twice as many users per server with faster times for all users". Which is pretty stunning, although it should be viewed as a condemnation of the current code, I'm afraid.

I had looked briefly at Andrea's implementation in the past. I will look again. I had thought that this approach did not get much traction.

Some quick thoughts about this approach though.

- It is not a proportional weight controller. It is more of limiting bandwidth in absolute numbers for each cgroup on each disk.

So each cgroup will define a rule for each disk in the system mentioning at what maximum rate that cgroup can issue IO to that disk and throttle the IO from that cgroup if rate has excedded.

Above requirement can create configuration problems.

- If there are large number of disks in system, per cgroup one shall have to create rules for each disk. Until and unless admin knows what applications are in which cgroup and strictly what disk these applications do IO to and create rules for only those disks.

- I think problem gets compounded if there is a hierarchy of logical devices. I think in that case one shall have to create rules for logical devices and not actual physical devices.

- Because it is not proportional weight distribution, if some cgroup is not using its planned BW, other group sharing the disk can not make use of spare BW.

- I think one should know in advance the throughput rate of underlying media and also know competing applications so that one can statically define the BW assigned to each cgroup on each disk.

This will be difficult. Effective BW extracted out of a rotational media is dependent on the seek pattern so one shall have to either try to make some conservative estimates and try to divide BW (we will not utilize disk fully) or take some peak numbers and divide BW (cgroup might not get the maximum rate configured).

- Above problems will comound when one goes for deeper hierarhical configurations.

I think for renewable resources like disk time, it might be a good idea to do a proportional weight controller to ensure fairness at the same time achive best throughput possible.

Andrea, please correct me if I have misunderstood the things.

Quoted 3 lines

So my question is: what is the definitive list of proposed-io-controller-implementations and how do I cunningly get all you guys to check each others homework? :)

I will try to summarize some of the proposals I am aware of.

- Elevator/IO scheduler modification based IO controllers
- This proposal
- cfq io scheduler based control (Satoshi Uchida, NEC)
- One more cfq based io control (vasily, OpenVZ)
- AS io scheduler based control (Naveen Gupta, Google)

- Io-throttling (Andrea Righi)
- Max Bandwidth Controller

- dm-ioband (valinux)
- Proportional weight IO controller.

- Generic IO controller (Vivek Goyal, RedHat)
- My initial attempt to do proportional division of amount of bio per cgroup at request queue level. This was inspired from dm-ioband.

I think this proposal should hopefully meet the requirements as envisoned by other elevator based IO controller solutions.

dm-ioband
---------
I have briefly looked at dm-ioband also and following were some of the concerns I had raised in the past.

- Need of a dm device for every device we want to control

- This requirement looks odd. It forces everybody to use dm-tools and if there are lots of disks in the system, configuation is pain.

- It does not support hiearhical grouping.

- Possibly can break the assumptions of underlying IO schedulers.

- There is no notion of task classes. So tasks of all the classes are at same level from resource contention point of view. The only thing which differentiates them is cgroup weight. Which does not answer the question that an RT task or RT cgroup should starve the peer cgroup if need be as RT cgroup should get priority access.

- Because of FIFO release of buffered bios, it is possible that task of lower priority gets more IO done than the task of higher priority.

- Buffering at multiple levels and FIFO dispatch can have more interesting hard to solve issues.

- Assume there is sequential reader and an aggressive writer in the cgroup. It might happen that writer pushed lot of write requests in the FIFO queue first and then a read request from reader comes. Now it might happen that cfq does not see this read request for a long time (if cgroup weight is less) and this writer will starve the reader in this cgroup.

Even cfq anticipation logic will not help here because when that first read request actually gets to cfq, cfq might choose to idle for more read requests to come, but the agreesive writer might have again flooded the FIFO queue in the group and cfq will not see subsequent read request for a long time and will unnecessarily idle for read.

- Task grouping logic
- We already have the notion of cgroup where tasks can be grouped in hierarhical manner. dm-ioband does not make full use of that and comes up with own mechansim of grouping tasks (apart from cgroup). And there are odd ways of specifying cgroup id while configuring the dm-ioband device.

IMHO, once somebody has created the cgroup hieararchy, any IO controller logic should be able to internally read that hiearchy and provide control. There should not be need of any other configuration utity on top of cgroup.

My RFC patches had tried to get rid of this external configuration requirement.

- Task and Groups can not be treated at same level.

- Because at any second level solution we are controlling bio per cgroup and don't have any notion of which task queue bio belongs to, one can not treat task and group at same level.

What I meant is following.

root
/ | \
1 2 A
/ \
3 4

In dm-ioband approach, at top level tasks 1 and 2 will get 50% of BW together and group A will get 50%. Ideally along the lines of cpu controller, I would expect it to be 33% each for task 1 task 2 and group A.

This can create interesting scenarios where assumg task1 is an RT class task. Now one would expect task 1 get all the BW possible starving task 2 and group A, but that will not be the case and task1 will get 50% of BW.

Not that it is critically important but it would probably be nice if we can maitain same semantics as cpu controller. In elevator layer solution we can do it at least for CFQ scheduler as it maintains separate io queue per io context.

This is in general an issue for any 2nd level IO controller which only accounts for io groups and not for io queues per process.

- We will end copying a lot of code/logic from cfq

- To address many of the concerns like multi class scheduler we will end up duplicating code of IO scheduler. Why can't we have a one point hierarchical IO scheduling (This patchset).
Thanks
Vivek

#6 lore raw

Re: [PATCH 01/10] Documentation

In reply to Andrew Morton

Hi Andrew,

Quoted 6 lines

Also.. there are so many IO controller implementations that I've lost track of who is doing what. I do have one private report here that Andreas's controller "is incredibly productive for us and has allowed us to put twice as many users per server with faster times for all users". Which is pretty stunning, although it should be viewed as a condemnation of the current code, I'm afraid.

I'm developing dm-ioband, which is an another IO controller, and I would like to hear your comments about dm-ioband if you have.

dm-ioband web page:
http://people.valinux.co.jp/~ryov/dm-ioband/

Quoted 3 lines

So my question is: what is the definitive list of proposed-io-controller-implementations and how do I cunningly get all you guys to check each others homework? :)

Dm-ioband is implemented as a device-mapper driver, so I'm proposing dm-ioband to dm-devel and I hope device-mapper staffs take care of it.

Thanks,
Ryo Tsuruta

#7 lore raw

Re: [PATCH 01/10] Documentation

In reply to Andrew Morton
Quoted 9 lines · Andrew Morton

Documentation/block/io-controller.txt line 114

Quoted 4 lines · On Wed, 11 Mar 2009 21:56:46 -0400 Vivek Goyal <[email protected]> wrote:
Documentation/block/io-controller.txt new file +4 -0
+Currently "current" task
+is used to determine the cgroup (hence io group) of the request. Down the
+line we need to make use of bio-cgroup patches to map delayed writes to
+right group.

You handled this problem pretty neatly!

Quoted 8 lines

It's always been a BIG problem for all the io-controlling schemes, and most of them seem to have "handled" it in the above way :(

But for many workloads, writeback is the majority of the IO and it has always been the form of IO which has caused us the worst contention and latency problems. So I don't think that we can proceed with _anything_ until we at least have a convincing plan here.

Hi Andrew,

Nauman is already maintaining the bio-cgroup patches (originally from valinux folks) on top of this patchset for attributing write requests to correct cgroup. We did not include those in initial posting thinking that patchest will bloat further.

We can pull in bio-cgroup patches also in this series to attribute writes to right cgroup.

Quoted 8 lines

Also.. there are so many IO controller implementations that I've lost track of who is doing what. I do have one private report here that Andreas's controller "is incredibly productive for us and has allowed us to put twice as many users per server with faster times for all users". Which is pretty stunning, although it should be viewed as a condemnation of the current code, I'm afraid.

I had looked briefly at Andrea's implementation in the past. I will look again. I had thought that this approach did not get much traction.

Some quick thoughts about this approach though.

- It is not a proportional weight controller. It is more of limiting bandwidth in absolute numbers for each cgroup on each disk.

So each cgroup will define a rule for each disk in the system mentioning at what maximum rate that cgroup can issue IO to that disk and throttle the IO from that cgroup if rate has excedded.

Above requirement can create configuration problems.

- If there are large number of disks in system, per cgroup one shall have to create rules for each disk. Until and unless admin knows what applications are in which cgroup and strictly what disk these applications do IO to and create rules for only those disks.

- I think problem gets compounded if there is a hierarchy of logical devices. I think in that case one shall have to create rules for logical devices and not actual physical devices.

- Because it is not proportional weight distribution, if some cgroup is not using its planned BW, other group sharing the disk can not make use of spare BW.

- I think one should know in advance the throughput rate of underlying media and also know competing applications so that one can statically define the BW assigned to each cgroup on each disk.

This will be difficult. Effective BW extracted out of a rotational media is dependent on the seek pattern so one shall have to either try to make some conservative estimates and try to divide BW (we will not utilize disk fully) or take some peak numbers and divide BW (cgroup might not get the maximum rate configured).

- Above problems will comound when one goes for deeper hierarhical configurations.

I think for renewable resources like disk time, it might be a good idea to do a proportional weight controller to ensure fairness at the same time achive best throughput possible.

Andrea, please correct me if I have misunderstood the things.

Quoted 3 lines

So my question is: what is the definitive list of proposed-io-controller-implementations and how do I cunningly get all you guys to check each others homework? :)

I will try to summarize some of the proposals I am aware of.

- Elevator/IO scheduler modification based IO controllers
- This proposal
- cfq io scheduler based control (Satoshi Uchida, NEC)
- One more cfq based io control (vasily, OpenVZ)
- AS io scheduler based control (Naveen Gupta, Google)

- Io-throttling (Andrea Righi)
- Max Bandwidth Controller

- dm-ioband (valinux)
- Proportional weight IO controller.

- Generic IO controller (Vivek Goyal, RedHat)
- My initial attempt to do proportional division of amount of bio per cgroup at request queue level. This was inspired from dm-ioband.

I think this proposal should hopefully meet the requirements as envisoned by other elevator based IO controller solutions.

dm-ioband
---------
I have briefly looked at dm-ioband also and following were some of the concerns I had raised in the past.

- Need of a dm device for every device we want to control

- This requirement looks odd. It forces everybody to use dm-tools and if there are lots of disks in the system, configuation is pain.

- It does not support hiearhical grouping.

- Possibly can break the assumptions of underlying IO schedulers.

- There is no notion of task classes. So tasks of all the classes are at same level from resource contention point of view. The only thing which differentiates them is cgroup weight. Which does not answer the question that an RT task or RT cgroup should starve the peer cgroup if need be as RT cgroup should get priority access.

- Because of FIFO release of buffered bios, it is possible that task of lower priority gets more IO done than the task of higher priority.

- Buffering at multiple levels and FIFO dispatch can have more interesting hard to solve issues.

- Assume there is sequential reader and an aggressive writer in the cgroup. It might happen that writer pushed lot of write requests in the FIFO queue first and then a read request from reader comes. Now it might happen that cfq does not see this read request for a long time (if cgroup weight is less) and this writer will starve the reader in this cgroup.

Even cfq anticipation logic will not help here because when that first read request actually gets to cfq, cfq might choose to idle for more read requests to come, but the agreesive writer might have again flooded the FIFO queue in the group and cfq will not see subsequent read request for a long time and will unnecessarily idle for read.

- Task grouping logic
- We already have the notion of cgroup where tasks can be grouped in hierarhical manner. dm-ioband does not make full use of that and comes up with own mechansim of grouping tasks (apart from cgroup). And there are odd ways of specifying cgroup id while configuring the dm-ioband device.

IMHO, once somebody has created the cgroup hieararchy, any IO controller logic should be able to internally read that hiearchy and provide control. There should not be need of any other configuration utity on top of cgroup.

My RFC patches had tried to get rid of this external configuration requirement.

- Task and Groups can not be treated at same level.

- Because at any second level solution we are controlling bio per cgroup and don't have any notion of which task queue bio belongs to, one can not treat task and group at same level.

What I meant is following.

root
/ | \
1 2 A
/ \
3 4

In dm-ioband approach, at top level tasks 1 and 2 will get 50% of BW together and group A will get 50%. Ideally along the lines of cpu controller, I would expect it to be 33% each for task 1 task 2 and group A.

This can create interesting scenarios where assumg task1 is an RT class task. Now one would expect task 1 get all the BW possible starving task 2 and group A, but that will not be the case and task1 will get 50% of BW.

Not that it is critically important but it would probably be nice if we can maitain same semantics as cpu controller. In elevator layer solution we can do it at least for CFQ scheduler as it maintains separate io queue per io context.

This is in general an issue for any 2nd level IO controller which only accounts for io groups and not for io queues per process.

- We will end copying a lot of code/logic from cfq

- To address many of the concerns like multi class scheduler we will end up duplicating code of IO scheduler. Why can't we have a one point hierarchical IO scheduling (This patchset).
Thanks
Vivek

A message that is not in the archive (sent off-list or not yet received): [email protected]

#9 lore raw

Re: [PATCH 01/10] Documentation

Hi Vivek,

Quoted 10 lines

dm-ioband
---------
I have briefly looked at dm-ioband also and following were some of the concerns I had raised in the past.

- Need of a dm device for every device we want to control

- This requirement looks odd. It forces everybody to use dm-tools and if there are lots of disks in the system, configuation is pain.

I don't think it's a pain. Could it be easily done by writing a small script?

Quoted 1 line

- It does not support hiearhical grouping.

I can implement hierarchical grouping to dm-ioband if it's really necessary, but at this point, I don't think it's really necessary and I want to keep the code simple.

Quoted 24 lines

- Possibly can break the assumptions of underlying IO schedulers.

- There is no notion of task classes. So tasks of all the classes are at same level from resource contention point of view. The only thing which differentiates them is cgroup weight. Which does not answer the question that an RT task or RT cgroup should starve the peer cgroup if need be as RT cgroup should get priority access.

- Because of FIFO release of buffered bios, it is possible that task of lower priority gets more IO done than the task of higher priority.

- Buffering at multiple levels and FIFO dispatch can have more interesting hard to solve issues.

- Assume there is sequential reader and an aggressive writer in the cgroup. It might happen that writer pushed lot of write requests in the FIFO queue first and then a read request from reader comes. Now it might happen that cfq does not see this read request for a long time (if cgroup weight is less) and this writer will starve the reader in this cgroup.

Quoted 6 lines

Even cfq anticipation logic will not help here because when that first read request actually gets to cfq, cfq might choose to idle for more read requests to come, but the agreesive writer might have again flooded the FIFO queue in the group and cfq will not see subsequent read request for a long time and will unnecessarily idle for read.

I think it's just a matter of which you prioritize, bandwidth or io-class. What do you do when the RT task issues a lot of I/O?

Quoted 7 lines

- Task grouping logic
- We already have the notion of cgroup where tasks can be grouped in hierarhical manner. dm-ioband does not make full use of that and comes up with own mechansim of grouping tasks (apart from cgroup). And there are odd ways of specifying cgroup id while configuring the dm-ioband device.

Quoted 7 lines

IMHO, once somebody has created the cgroup hieararchy, any IO controller logic should be able to internally read that hiearchy and provide control. There should not be need of any other configuration utity on top of cgroup.

My RFC patches had tried to get rid of this external configuration requirement.

The reason is that it makes bio-cgroup easy to use for dm-ioband. But It's not a final design of the interface between dm-ioband and cgroup.

Quoted 24 lines

- Task and Groups can not be treated at same level.

- Because at any second level solution we are controlling bio per cgroup and don't have any notion of which task queue bio belongs to, one can not treat task and group at same level.

What I meant is following.

root
/ | \
1 2 A
/ \
3 4

In dm-ioband approach, at top level tasks 1 and 2 will get 50% of BW together and group A will get 50%. Ideally along the lines of cpu controller, I would expect it to be 33% each for task 1 task 2 and group A.

This can create interesting scenarios where assumg task1 is an RT class task. Now one would expect task 1 get all the BW possible starving task 2 and group A, but that will not be the case and task1 will get 50% of BW.

Quoted 4 lines

Not that it is critically important but it would probably be nice if we can maitain same semantics as cpu controller. In elevator layer solution we can do it at least for CFQ scheduler as it maintains separate io queue per io context.

I will consider following the CPU controller's manner when dm-ioband supports hierarchical grouping.

Quoted 10 lines

This is in general an issue for any 2nd level IO controller which only accounts for io groups and not for io queues per process.

- We will end copying a lot of code/logic from cfq

- To address many of the concerns like multi class scheduler we will end up duplicating code of IO scheduler. Why can't we have a one point hierarchical IO scheduling (This patchset).
Thanks
Vivek

Thanks,
Ryo Tsuruta

#10 lore raw

Re: [PATCH 01/10] Documentation

In reply to a message not in the archive
Quoted 2 lines · Vivek Goyal
Quoted 1 line · Andrew Morton

On Wed, 11 Mar 2009 21:56:46 -0400 Vivek Goyal <[email protected]> wrote:

[snip]

Quoted 10 lines
Quoted 7 lines

Also.. there are so many IO controller implementations that I've lost track of who is doing what. I do have one private report here that Andreas's controller "is incredibly productive for us and has allowed us to put twice as many users per server with faster times for all users". Which is pretty stunning, although it should be viewed as a condemnation of the current code, I'm afraid.

I had looked briefly at Andrea's implementation in the past. I will look again. I had thought that this approach did not get much traction.

Hi Vivek, sorry for my late reply. I periodically upload the latest versions of io-throttle here if you're still interested:
http://download.systemimager.org/~arighi/linux/patches/io-throttle/

There's no consistent changes respect to the latest version I posted to the LKML, just rebasing to the recent kernels.

Quoted 9 lines

Some quick thoughts about this approach though.

- It is not a proportional weight controller. It is more of limiting bandwidth in absolute numbers for each cgroup on each disk.

So each cgroup will define a rule for each disk in the system mentioning at what maximum rate that cgroup can issue IO to that disk and throttle the IO from that cgroup if rate has excedded.

Correct. Add also the proportional weight control has been in the TODO list since the early versions, but I never dedicated too much effort to implement this feature, I can focus on this and try to write something if we all think it is worth to be done.

Quoted 8 lines

Above requirement can create configuration problems.

- If there are large number of disks in system, per cgroup one shall have to create rules for each disk. Until and unless admin knows what applications are in which cgroup and strictly what disk these applications do IO to and create rules for only those disks.

I don't think this is a huge problem anyway. IMHO a userspace tool, e.g. a script, would be able to efficiently create/modify rules parsing user defined rules in some human-readable form (config files, etc.), even in presence of hundreds of disk. The same is valid for dm-ioband I think.

Quoted 4 lines

- I think problem gets compounded if there is a hierarchy of logical devices. I think in that case one shall have to create rules for logical devices and not actual physical devices.

With logical devices you mean device-mapper devices (i.e. LVM, software RAID, etc.)? or do you mean that we need to introduce the concept of "logical device" to easily (quickly) configure IO requirements and then map those logical devices to the actual physical devices? In this case I think this can be addressed in userspace. Or maybe I'm totally missing the point here.

Quoted 5 lines

- Because it is not proportional weight distribution, if some cgroup is not using its planned BW, other group sharing the disk can not make use of spare BW.

Right.

Quoted 9 lines

- I think one should know in advance the throughput rate of underlying media and also know competing applications so that one can statically define the BW assigned to each cgroup on each disk.

This will be difficult. Effective BW extracted out of a rotational media is dependent on the seek pattern so one shall have to either try to make some conservative estimates and try to divide BW (we will not utilize disk fully) or take some peak numbers and divide BW (cgroup might not get the maximum rate configured).

Correct. I think the proportional weight approach is the only solution to efficiently use the whole BW. OTOH absolute limiting rules offer a better control over QoS, because you can totally remove performance bursts/peaks that could break QoS requirements for short periods of time. So, my "ideal" IO controller should allow to define both rules:
absolute and proportional limits.

I still have to look closely at your patchset anyway. I will do and give a feedback.

-Andrea

#11 lore raw

Re: [PATCH 01/10] Documentation

In reply to Vivek Goyal

Hi Vivek,

Quoted 10 lines

dm-ioband
---------
I have briefly looked at dm-ioband also and following were some of the concerns I had raised in the past.

- Need of a dm device for every device we want to control

- This requirement looks odd. It forces everybody to use dm-tools and if there are lots of disks in the system, configuation is pain.

I don't think it's a pain. Could it be easily done by writing a small script?

Quoted 1 line

- It does not support hiearhical grouping.

I can implement hierarchical grouping to dm-ioband if it's really necessary, but at this point, I don't think it's really necessary and I want to keep the code simple.

Quoted 24 lines

- Possibly can break the assumptions of underlying IO schedulers.

- There is no notion of task classes. So tasks of all the classes are at same level from resource contention point of view. The only thing which differentiates them is cgroup weight. Which does not answer the question that an RT task or RT cgroup should starve the peer cgroup if need be as RT cgroup should get priority access.

- Because of FIFO release of buffered bios, it is possible that task of lower priority gets more IO done than the task of higher priority.

- Buffering at multiple levels and FIFO dispatch can have more interesting hard to solve issues.

- Assume there is sequential reader and an aggressive writer in the cgroup. It might happen that writer pushed lot of write requests in the FIFO queue first and then a read request from reader comes. Now it might happen that cfq does not see this read request for a long time (if cgroup weight is less) and this writer will starve the reader in this cgroup.

Quoted 6 lines

Even cfq anticipation logic will not help here because when that first read request actually gets to cfq, cfq might choose to idle for more read requests to come, but the agreesive writer might have again flooded the FIFO queue in the group and cfq will not see subsequent read request for a long time and will unnecessarily idle for read.

I think it's just a matter of which you prioritize, bandwidth or io-class. What do you do when the RT task issues a lot of I/O?

Quoted 7 lines

- Task grouping logic
- We already have the notion of cgroup where tasks can be grouped in hierarhical manner. dm-ioband does not make full use of that and comes up with own mechansim of grouping tasks (apart from cgroup). And there are odd ways of specifying cgroup id while configuring the dm-ioband device.

Quoted 7 lines

IMHO, once somebody has created the cgroup hieararchy, any IO controller logic should be able to internally read that hiearchy and provide control. There should not be need of any other configuration utity on top of cgroup.

My RFC patches had tried to get rid of this external configuration requirement.

The reason is that it makes bio-cgroup easy to use for dm-ioband. But It's not a final design of the interface between dm-ioband and cgroup.

Quoted 24 lines

- Task and Groups can not be treated at same level.

- Because at any second level solution we are controlling bio per cgroup and don't have any notion of which task queue bio belongs to, one can not treat task and group at same level.

What I meant is following.

root
/ | \
1 2 A
/ \
3 4

In dm-ioband approach, at top level tasks 1 and 2 will get 50% of BW together and group A will get 50%. Ideally along the lines of cpu controller, I would expect it to be 33% each for task 1 task 2 and group A.

This can create interesting scenarios where assumg task1 is an RT class task. Now one would expect task 1 get all the BW possible starving task 2 and group A, but that will not be the case and task1 will get 50% of BW.

Quoted 4 lines

Not that it is critically important but it would probably be nice if we can maitain same semantics as cpu controller. In elevator layer solution we can do it at least for CFQ scheduler as it maintains separate io queue per io context.

I will consider following the CPU controller's manner when dm-ioband supports hierarchical grouping.

Quoted 10 lines

This is in general an issue for any 2nd level IO controller which only accounts for io groups and not for io queues per process.

- We will end copying a lot of code/logic from cfq

- To address many of the concerns like multi class scheduler we will end up duplicating code of IO scheduler. Why can't we have a one point hierarchical IO scheduling (This patchset).
Thanks
Vivek

Thanks,
Ryo Tsuruta

A message that is not in the archive (sent off-list or not yet received): 20090316.174043.193698189.ryov-jCdQPDEk3idL9jVzuh4AOg@public.gmane.org

#13 lore raw

Re: [PATCH 01/10] Documentation

Quoted 7 lines · Ryo Tsuruta

Hi Vivek,

Quoted 5 lines

dm-ioband
---------
I have briefly looked at dm-ioband also and following were some of the concerns I had raised in the past.

Quoted 9 lines
Quoted 5 lines

- Need of a dm device for every device we want to control

- This requirement looks odd. It forces everybody to use dm-tools and if there are lots of disks in the system, configuation is pain.

I don't think it's a pain. Could it be easily done by writing a small script?

I think it is an extra hassle which can be avoided. Following are some of the thoughts about configuration and issues. Looking at these, IMHO, it is not simple to configure dm-ioband.

- So if there are 100 disks in a system, and lets say 5 partitions on each disk, then script needs to create a dm-ioband device for every partition. So I will end up creating 500 dm-ioband devices. This is not taking into picture the dm-ioband devices people might end up creating on intermediate logical nodes.

- Need of dm tools to create devices and create groups.

- I am look at dm-ioband help on web and thinking are these commands really simple and hassle free for a user who does not use dm in his setup.

For two dm-ioband device creations on two partitions.

 # echo "0 $(blockdev --getsize /dev/sda1) ioband /dev/sda1 1 0 0 none" \
     "weight 0 :40" | dmsetup create ioband1
 # echo "0 $(blockdev --getsize /dev/sda2) ioband /dev/sda2 1 0 0 none" \
     "weight 0 :10" | dmsetup create ioband2 

- Following are the commands just to create two groups on a single io-band device.

 # dmsetup message ioband1 0 type user
 # dmsetup message ioband1 0 attach 1000
 # dmsetup message ioband1 0 attach 2000
 # dmsetup message ioband1 0 weight 1000:30
 # dmsetup message ioband1 0 weight 2000:20

Now think of a decent size group hierarchy (say 50 groups) on 500 ioband device system. So that would be 50*500 = 25000 group creation commands.

- So if an admin wants to group applications using cgroup, first he needs to create cgroup hierarchy. Then he needs to take all the cgroup ids and provide these to this dm-ioband device with the help of dmsetup command.

dmsetup message ioband1 0 attach <cgroup id>

cgroup has already provided us nice grouping facility in hierarchical manner. This extra step is cumbersome and completely unnecessary.

- These configuration commands will become even much more complicated once you start supporting hierachical setup. All the hierarchy information shall have to passed in the command itself in one way or other once a group is being created.

- You will be limited in terms of functionlity. I am assuming these group creation operations will be limited to "root" user. A very common requiremnt we are seeming now a days is that admin will create a top level cgroup and then let user create/manage more groups with-in top level group.

For example.

			  root
			/  |  \
		     u1    u2 others 		      

Here u1 and u2 are two different users on the system. Here admin can create top level cgroups for users and assign users weight from IO point of view. Now individual users should be able to create groups of their own and manage their tasks. Cgroup infrastructure allows all this.

In the above setup it will become very very hard to let user also create its own groups in top level group. You shall have to keep all the information which filesystem keeps in terms of file permissions etc.

So IMHO, configuration of dm-ioband devices and groups is complicated and it can be simplified a lot. Secondly, it does not seem to be a good idea to not make use of cgroup infrastrucuture and come up own ways of grouping things.

Quoted 6 lines
Quoted 1 line

- It does not support hiearhical grouping.

I can implement hierarchical grouping to dm-ioband if it's really necessary, but at this point, I don't think it's really necessary and I want to keep the code simple.

We do need hierarchical support.

In fact later in the mail you have specified that you will consider treating task and groups at same level. The moment you do that, one flat hiearchy will mean a single "root" group only and no groups with-in that. Until and unless you implement hiearchical support you can't create even single level of groups with-in "root".

Secondly, i think dm-ioband will become very complex (especially in terms of managing configuration), the moment hiearchical support is introduced. So it would be a good idea to implement the hiearchical support now and get to know the full complexity of the system.

Quoted 31 lines
Quoted 24 lines

- Possibly can break the assumptions of underlying IO schedulers.

- There is no notion of task classes. So tasks of all the classes are at same level from resource contention point of view. The only thing which differentiates them is cgroup weight. Which does not answer the question that an RT task or RT cgroup should starve the peer cgroup if need be as RT cgroup should get priority access.

- Because of FIFO release of buffered bios, it is possible that task of lower priority gets more IO done than the task of higher priority.

- Buffering at multiple levels and FIFO dispatch can have more interesting hard to solve issues.

- Assume there is sequential reader and an aggressive writer in the cgroup. It might happen that writer pushed lot of write requests in the FIFO queue first and then a read request from reader comes. Now it might happen that cfq does not see this read request for a long time (if cgroup weight is less) and this writer will starve the reader in this cgroup.

Quoted 6 lines

Even cfq anticipation logic will not help here because when that first read request actually gets to cfq, cfq might choose to idle for more read requests to come, but the agreesive writer might have again flooded the FIFO queue in the group and cfq will not see subsequent read request for a long time and will unnecessarily idle for read.

Quoted 3 lines

I think it's just a matter of which you prioritize, bandwidth or io-class. What do you do when the RT task issues a lot of I/O?

This is a multi-class scheduler. We first prioritize class and then handle tasks with-in class. So RT class will always get to dispatch first and can starve Best effort class tasks if it is issueing lots of IO.

You just don't have any notion of RT groups. So if admin wants to make sure that and RT tasks always gets the disk access first, there is no way to ensure that. The best thing in this setup one can do is assign higher weight to RT task group. This group will still be doing proportional weight scheduling with Best effort class groups or Idle task groups. That's not multi-class scheduling is.

So in your patches there is no differentiation between classes. A best effort task is competing equally hard as RT task. For example.

			root
		 	/  \
		   RT task  Group (best effort class)
				/ \
			       T1  T2 

Here T1 and T2 are best effort class tasks and they are sharing disk bandwidth with RT task. Instead, RT task should get exclusive access to disk.

Secondly, two of the above issues I have mentioned are for tasks with-in same class and how FIFO dispatch will create the problems. These are problems with any second level controller. These will be really hard to solve the issues and will force us to copy more code from cfq and other subsystems.

Quoted 12 lines
Quoted 12 lines

- Task grouping logic
- We already have the notion of cgroup where tasks can be grouped in hierarhical manner. dm-ioband does not make full use of that and comes up with own mechansim of grouping tasks (apart from cgroup). And there are odd ways of specifying cgroup id while configuring the dm-ioband device.

IMHO, once somebody has created the cgroup hieararchy, any IO controller logic should be able to internally read that hiearchy and provide control. There should not be need of any other configuration utity on top of cgroup.

Quoted 6 lines
Quoted 2 lines

My RFC patches had tried to get rid of this external configuration requirement.

The reason is that it makes bio-cgroup easy to use for dm-ioband. But It's not a final design of the interface between dm-ioband and cgroup.

It makes it easy for dm-ioband implementation but harder for the user.

What is the alternate interface?

Quoted 25 lines
Quoted 25 lines

- Task and Groups can not be treated at same level.

- Because at any second level solution we are controlling bio per cgroup and don't have any notion of which task queue bio belongs to, one can not treat task and group at same level.

What I meant is following.

root
/ | \
1 2 A
/ \
3 4

In dm-ioband approach, at top level tasks 1 and 2 will get 50% of BW together and group A will get 50%. Ideally along the lines of cpu controller, I would expect it to be 33% each for task 1 task 2 and group A.

This can create interesting scenarios where assumg task1 is an RT class task. Now one would expect task 1 get all the BW possible starving task 2 and group A, but that will not be the case and task1 will get 50% of BW.

Quoted 7 lines
Quoted 4 lines

Not that it is critically important but it would probably be nice if we can maitain same semantics as cpu controller. In elevator layer solution we can do it at least for CFQ scheduler as it maintains separate io queue per io context.

I will consider following the CPU controller's manner when dm-ioband supports hierarchical grouping.

But this is an issue even now. If you want to consider task and group at the same level, then you will end up creating separate queues for all the tasks (and not only queues for groups). This will essentially become CFQ.

Quoted 9 lines
Quoted 9 lines

This is in general an issue for any 2nd level IO controller which only accounts for io groups and not for io queues per process.

- We will end copying a lot of code/logic from cfq

- To address many of the concerns like multi class scheduler we will end up duplicating code of IO scheduler. Why can't we have a one point hierarchical IO scheduling (This patchset).

More details about this point.

- To make dm-ioband support multiclass task/groups, we will end up inheriting logic from cfq/bfq.

- To treat task and group at same level we will end up creating separate queues for each task and then import lots of cfq/bfq logic for managing those queues.

- The moment we move to hiearchical support, you will end up creating equivalent logic of our patches.

The point is, why to do all this? CFQ has already solved the problem of multi class IO scheduler and providing service differentiation between tasks of different priority. With cgroup stuff, we need to just extend existing CFQ logic so that it supports hiearchical scheduling and we will have a good IO controller in place.

Can you please point out specifically why do you think extending CFQ logic to support hiearchical scheduling and sharing code with other IO schedulers is not a good idea to implement hiearchical IO control?

Thanks
Vivek

#14 lore raw

Re: [PATCH 01/10] Documentation

In reply to Ryo Tsuruta
Quoted 7 lines · Ryo Tsuruta

Hi Vivek,

Quoted 5 lines

dm-ioband
---------
I have briefly looked at dm-ioband also and following were some of the concerns I had raised in the past.

Quoted 9 lines
Quoted 5 lines

- Need of a dm device for every device we want to control

- This requirement looks odd. It forces everybody to use dm-tools and if there are lots of disks in the system, configuation is pain.

I don't think it's a pain. Could it be easily done by writing a small script?

I think it is an extra hassle which can be avoided. Following are some of the thoughts about configuration and issues. Looking at these, IMHO, it is not simple to configure dm-ioband.

- So if there are 100 disks in a system, and lets say 5 partitions on each disk, then script needs to create a dm-ioband device for every partition. So I will end up creating 500 dm-ioband devices. This is not taking into picture the dm-ioband devices people might end up creating on intermediate logical nodes.

- Need of dm tools to create devices and create groups.

- I am look at dm-ioband help on web and thinking are these commands really simple and hassle free for a user who does not use dm in his setup.

For two dm-ioband device creations on two partitions.

 # echo "0 $(blockdev --getsize /dev/sda1) ioband /dev/sda1 1 0 0 none" \
     "weight 0 :40" | dmsetup create ioband1
 # echo "0 $(blockdev --getsize /dev/sda2) ioband /dev/sda2 1 0 0 none" \
     "weight 0 :10" | dmsetup create ioband2 

- Following are the commands just to create two groups on a single io-band device.

 # dmsetup message ioband1 0 type user
 # dmsetup message ioband1 0 attach 1000
 # dmsetup message ioband1 0 attach 2000
 # dmsetup message ioband1 0 weight 1000:30
 # dmsetup message ioband1 0 weight 2000:20

Now think of a decent size group hierarchy (say 50 groups) on 500 ioband device system. So that would be 50*500 = 25000 group creation commands.

- So if an admin wants to group applications using cgroup, first he needs to create cgroup hierarchy. Then he needs to take all the cgroup ids and provide these to this dm-ioband device with the help of dmsetup command.

dmsetup message ioband1 0 attach <cgroup id>

cgroup has already provided us nice grouping facility in hierarchical manner. This extra step is cumbersome and completely unnecessary.

- These configuration commands will become even much more complicated once you start supporting hierachical setup. All the hierarchy information shall have to passed in the command itself in one way or other once a group is being created.

- You will be limited in terms of functionlity. I am assuming these group creation operations will be limited to "root" user. A very common requiremnt we are seeming now a days is that admin will create a top level cgroup and then let user create/manage more groups with-in top level group.

For example.

			  root
			/  |  \
		     u1    u2 others 		      

Here u1 and u2 are two different users on the system. Here admin can create top level cgroups for users and assign users weight from IO point of view. Now individual users should be able to create groups of their own and manage their tasks. Cgroup infrastructure allows all this.

In the above setup it will become very very hard to let user also create its own groups in top level group. You shall have to keep all the information which filesystem keeps in terms of file permissions etc.

So IMHO, configuration of dm-ioband devices and groups is complicated and it can be simplified a lot. Secondly, it does not seem to be a good idea to not make use of cgroup infrastrucuture and come up own ways of grouping things.

Quoted 6 lines
Quoted 1 line

- It does not support hiearhical grouping.

I can implement hierarchical grouping to dm-ioband if it's really necessary, but at this point, I don't think it's really necessary and I want to keep the code simple.

We do need hierarchical support.

In fact later in the mail you have specified that you will consider treating task and groups at same level. The moment you do that, one flat hiearchy will mean a single "root" group only and no groups with-in that. Until and unless you implement hiearchical support you can't create even single level of groups with-in "root".

Secondly, i think dm-ioband will become very complex (especially in terms of managing configuration), the moment hiearchical support is introduced. So it would be a good idea to implement the hiearchical support now and get to know the full complexity of the system.

Quoted 31 lines
Quoted 24 lines

- Possibly can break the assumptions of underlying IO schedulers.

- There is no notion of task classes. So tasks of all the classes are at same level from resource contention point of view. The only thing which differentiates them is cgroup weight. Which does not answer the question that an RT task or RT cgroup should starve the peer cgroup if need be as RT cgroup should get priority access.

- Because of FIFO release of buffered bios, it is possible that task of lower priority gets more IO done than the task of higher priority.

- Buffering at multiple levels and FIFO dispatch can have more interesting hard to solve issues.

- Assume there is sequential reader and an aggressive writer in the cgroup. It might happen that writer pushed lot of write requests in the FIFO queue first and then a read request from reader comes. Now it might happen that cfq does not see this read request for a long time (if cgroup weight is less) and this writer will starve the reader in this cgroup.

Quoted 6 lines

Even cfq anticipation logic will not help here because when that first read request actually gets to cfq, cfq might choose to idle for more read requests to come, but the agreesive writer might have again flooded the FIFO queue in the group and cfq will not see subsequent read request for a long time and will unnecessarily idle for read.

Quoted 3 lines

I think it's just a matter of which you prioritize, bandwidth or io-class. What do you do when the RT task issues a lot of I/O?

This is a multi-class scheduler. We first prioritize class and then handle tasks with-in class. So RT class will always get to dispatch first and can starve Best effort class tasks if it is issueing lots of IO.

You just don't have any notion of RT groups. So if admin wants to make sure that and RT tasks always gets the disk access first, there is no way to ensure that. The best thing in this setup one can do is assign higher weight to RT task group. This group will still be doing proportional weight scheduling with Best effort class groups or Idle task groups. That's not multi-class scheduling is.

So in your patches there is no differentiation between classes. A best effort task is competing equally hard as RT task. For example.

			root
		 	/  \
		   RT task  Group (best effort class)
				/ \
			       T1  T2 

Here T1 and T2 are best effort class tasks and they are sharing disk bandwidth with RT task. Instead, RT task should get exclusive access to disk.

Secondly, two of the above issues I have mentioned are for tasks with-in same class and how FIFO dispatch will create the problems. These are problems with any second level controller. These will be really hard to solve the issues and will force us to copy more code from cfq and other subsystems.

Quoted 12 lines
Quoted 12 lines

- Task grouping logic
- We already have the notion of cgroup where tasks can be grouped in hierarhical manner. dm-ioband does not make full use of that and comes up with own mechansim of grouping tasks (apart from cgroup). And there are odd ways of specifying cgroup id while configuring the dm-ioband device.

IMHO, once somebody has created the cgroup hieararchy, any IO controller logic should be able to internally read that hiearchy and provide control. There should not be need of any other configuration utity on top of cgroup.

Quoted 6 lines
Quoted 2 lines

My RFC patches had tried to get rid of this external configuration requirement.

The reason is that it makes bio-cgroup easy to use for dm-ioband. But It's not a final design of the interface between dm-ioband and cgroup.

It makes it easy for dm-ioband implementation but harder for the user.

What is the alternate interface?

Quoted 25 lines
Quoted 25 lines

- Task and Groups can not be treated at same level.

- Because at any second level solution we are controlling bio per cgroup and don't have any notion of which task queue bio belongs to, one can not treat task and group at same level.

What I meant is following.

root
/ | \
1 2 A
/ \
3 4

In dm-ioband approach, at top level tasks 1 and 2 will get 50% of BW together and group A will get 50%. Ideally along the lines of cpu controller, I would expect it to be 33% each for task 1 task 2 and group A.

This can create interesting scenarios where assumg task1 is an RT class task. Now one would expect task 1 get all the BW possible starving task 2 and group A, but that will not be the case and task1 will get 50% of BW.

Quoted 7 lines
Quoted 4 lines

Not that it is critically important but it would probably be nice if we can maitain same semantics as cpu controller. In elevator layer solution we can do it at least for CFQ scheduler as it maintains separate io queue per io context.

I will consider following the CPU controller's manner when dm-ioband supports hierarchical grouping.

But this is an issue even now. If you want to consider task and group at the same level, then you will end up creating separate queues for all the tasks (and not only queues for groups). This will essentially become CFQ.

Quoted 9 lines
Quoted 9 lines

This is in general an issue for any 2nd level IO controller which only accounts for io groups and not for io queues per process.

- We will end copying a lot of code/logic from cfq

- To address many of the concerns like multi class scheduler we will end up duplicating code of IO scheduler. Why can't we have a one point hierarchical IO scheduling (This patchset).

More details about this point.

- To make dm-ioband support multiclass task/groups, we will end up inheriting logic from cfq/bfq.

- To treat task and group at same level we will end up creating separate queues for each task and then import lots of cfq/bfq logic for managing those queues.

- The moment we move to hiearchical support, you will end up creating equivalent logic of our patches.

The point is, why to do all this? CFQ has already solved the problem of multi class IO scheduler and providing service differentiation between tasks of different priority. With cgroup stuff, we need to just extend existing CFQ logic so that it supports hiearchical scheduling and we will have a good IO controller in place.

Can you please point out specifically why do you think extending CFQ logic to support hiearchical scheduling and sharing code with other IO schedulers is not a good idea to implement hiearchical IO control?

Thanks
Vivek

#15 lore raw

Re: [PATCH 01/10] Documentation

In reply to Vivek Goyal
Quoted 2 lines · Vivek Goyal
Quoted 1 line · Andrew Morton

On Wed, 11 Mar 2009 21:56:46 -0400 Vivek Goyal <[email protected]> wrote:

[snip]

Quoted 10 lines
Quoted 7 lines

Also.. there are so many IO controller implementations that I've lost track of who is doing what. I do have one private report here that Andreas's controller "is incredibly productive for us and has allowed us to put twice as many users per server with faster times for all users". Which is pretty stunning, although it should be viewed as a condemnation of the current code, I'm afraid.

I had looked briefly at Andrea's implementation in the past. I will look again. I had thought that this approach did not get much traction.

Hi Vivek, sorry for my late reply. I periodically upload the latest versions of io-throttle here if you're still interested:
http://download.systemimager.org/~arighi/linux/patches/io-throttle/

There's no consistent changes respect to the latest version I posted to the LKML, just rebasing to the recent kernels.

Quoted 9 lines

Some quick thoughts about this approach though.

- It is not a proportional weight controller. It is more of limiting bandwidth in absolute numbers for each cgroup on each disk.

So each cgroup will define a rule for each disk in the system mentioning at what maximum rate that cgroup can issue IO to that disk and throttle the IO from that cgroup if rate has excedded.

Correct. Add also the proportional weight control has been in the TODO list since the early versions, but I never dedicated too much effort to implement this feature, I can focus on this and try to write something if we all think it is worth to be done.

Quoted 8 lines

Above requirement can create configuration problems.

- If there are large number of disks in system, per cgroup one shall have to create rules for each disk. Until and unless admin knows what applications are in which cgroup and strictly what disk these applications do IO to and create rules for only those disks.

I don't think this is a huge problem anyway. IMHO a userspace tool, e.g. a script, would be able to efficiently create/modify rules parsing user defined rules in some human-readable form (config files, etc.), even in presence of hundreds of disk. The same is valid for dm-ioband I think.

Quoted 4 lines

- I think problem gets compounded if there is a hierarchy of logical devices. I think in that case one shall have to create rules for logical devices and not actual physical devices.

With logical devices you mean device-mapper devices (i.e. LVM, software RAID, etc.)? or do you mean that we need to introduce the concept of "logical device" to easily (quickly) configure IO requirements and then map those logical devices to the actual physical devices? In this case I think this can be addressed in userspace. Or maybe I'm totally missing the point here.

Quoted 5 lines

- Because it is not proportional weight distribution, if some cgroup is not using its planned BW, other group sharing the disk can not make use of spare BW.

Right.

Quoted 9 lines

- I think one should know in advance the throughput rate of underlying media and also know competing applications so that one can statically define the BW assigned to each cgroup on each disk.

This will be difficult. Effective BW extracted out of a rotational media is dependent on the seek pattern so one shall have to either try to make some conservative estimates and try to divide BW (we will not utilize disk fully) or take some peak numbers and divide BW (cgroup might not get the maximum rate configured).

Correct. I think the proportional weight approach is the only solution to efficiently use the whole BW. OTOH absolute limiting rules offer a better control over QoS, because you can totally remove performance bursts/peaks that could break QoS requirements for short periods of time. So, my "ideal" IO controller should allow to define both rules:
absolute and proportional limits.

I still have to look closely at your patchset anyway. I will do and give a feedback.

-Andrea

A message that is not in the archive (sent off-list or not yet received): [email protected]

#17 lore raw

Re: [PATCH 01/10] Documentation

Quoted 81 lines · Andrea Righi
Quoted 2 lines · Vivek Goyal
Quoted 1 line · Andrew Morton

On Wed, 11 Mar 2009 21:56:46 -0400 Vivek Goyal <[email protected]> wrote:

[snip]

Quoted 10 lines
Quoted 7 lines

Also..  there are so many IO controller implementations that I've lost track of who is doing what.  I do have one private report here that Andreas's controller "is incredibly productive for us and has allowed us to put twice as many users per server with faster times for all users".  Which is pretty stunning, although it should be viewed as a condemnation of the current code, I'm afraid.

I had looked briefly at Andrea's implementation in the past. I will look again. I had thought that this approach did not get much traction.

Hi Vivek, sorry for my late reply. I periodically upload the latest versions of io-throttle here if you're still interested:
http://download.systemimager.org/~arighi/linux/patches/io-throttle/

There's no consistent changes respect to the latest version I posted to the LKML, just rebasing to the recent kernels.

Quoted 9 lines

Some quick thoughts about this approach though.

- It is not a proportional weight controller. It is more of limiting bandwidth in absolute numbers for each cgroup on each disk.

  So each cgroup will define a rule for each disk in the system mentioning at what maximum rate that cgroup can issue IO to that disk and throttle the IO from that cgroup if rate has excedded.

Correct. Add also the proportional weight control has been in the TODO list since the early versions, but I never dedicated too much effort to implement this feature, I can focus on this and try to write something if we all think it is worth to be done.

Quoted 8 lines

  Above requirement can create configuration problems.

      - If there are large number of disks in system, per cgroup one shall have to create rules for each disk. Until and unless admin knows what applications are in which cgroup and strictly what disk these applications do IO to and create rules for only those disks.

I don't think this is a huge problem anyway. IMHO a userspace tool, e.g. a script, would be able to efficiently create/modify rules parsing user defined rules in some human-readable form (config files, etc.), even in presence of hundreds of disk. The same is valid for dm-ioband I think.

Quoted 4 lines

      - I think problem gets compounded if there is a hierarchy of logical devices. I think in that case one shall have to create rules for logical devices and not actual physical devices.

With logical devices you mean device-mapper devices (i.e. LVM, software RAID, etc.)? or do you mean that we need to introduce the concept of "logical device" to easily (quickly) configure IO requirements and then map those logical devices to the actual physical devices? In this case I think this can be addressed in userspace. Or maybe I'm totally missing the point here.

Quoted 5 lines

- Because it is not proportional weight distribution, if some cgroup is not using its planned BW, other group sharing the disk can not make use of spare BW.

Right.

Quoted 9 lines

- I think one should know in advance the throughput rate of underlying media and also know competing applications so that one can statically define the BW assigned to each cgroup on each disk.

  This will be difficult. Effective BW extracted out of a rotational media is dependent on the seek pattern so one shall have to either try to make some conservative estimates and try to divide BW (we will not utilize disk fully) or take some peak numbers and divide BW (cgroup might not get the maximum rate configured).

Quoted 6 lines

Correct. I think the proportional weight approach is the only solution to efficiently use the whole BW. OTOH absolute limiting rules offer a better control over QoS, because you can totally remove performance bursts/peaks that could break QoS requirements for short periods of time. So, my "ideal" IO controller should allow to define both rules:
absolute and proportional limits.

I completely agree with Andrea here. The final solution has to have both absolute limits and proportions. But instead of adding a token based approach on top of proportional based system, I have been thinking about modifying the proportional approach to support absolute limits. This might not work, but I think this is an interesting idea to think about.

Here are my thoughts on it so far. We start with the patches that Vivek has sent, and change the notion of weights to percent. That is, the user space specifies percents of disk times instead of weights. We do not put entities in idle tree; whenever an entity is not backlogged, we still keep them in the active trees, and allocate them time slices. But since they have not requests in them, no requests will get dispatched during the time slices allocated to these entities. Moreover, if the percents of all entities do not add upto hundred, we introduce a dummy entity at each level to soak up the rest of "percent". This dummy entity would get time slices just like any other entity, but will not dispatch any requests. With these modification, we can limit entities to their allocated percent of disk time.

We might want a situation in which we want to allow certain entities to exceed their "percent" of disk time, while others should be limited. In this case, we can extend the above mentioned approach by introducing a secondary active tree. All entities which are allowed to exceed their "percent" can be queued in the secondary tree, besides the primary tree. Whenever an idle (or dummy) entity gets a time slice, instead of idling the disk, an entity can be picked from the secondary tree.

The advantage of an approach like this is that it will be relatively smaller modification of the proposed proportional approach. Moreover, entities will be throttled by getting time slices less frequently, instead of being allowed to send a burst and then getting starved (like in ticket based approach). The downside is that this approach sounds unconventional and probably have not been tried in other domains either. Thoughts? opinions?

I will create patches based on the above idea in a few weeks.

Quoted 6 lines

I still have to look closely at your patchset anyway. I will do and give a feedback.

-Andrea

#18 lore raw

Re: [PATCH 01/10] Documentation

In reply to Andrea Righi
Quoted 81 lines · Andrea Righi
Quoted 2 lines · Vivek Goyal
Quoted 1 line · Andrew Morton

On Wed, 11 Mar 2009 21:56:46 -0400 Vivek Goyal <[email protected]> wrote:

[snip]

Quoted 10 lines
Quoted 7 lines

Also..  there are so many IO controller implementations that I've lost track of who is doing what.  I do have one private report here that Andreas's controller "is incredibly productive for us and has allowed us to put twice as many users per server with faster times for all users".  Which is pretty stunning, although it should be viewed as a condemnation of the current code, I'm afraid.

I had looked briefly at Andrea's implementation in the past. I will look again. I had thought that this approach did not get much traction.

Hi Vivek, sorry for my late reply. I periodically upload the latest versions of io-throttle here if you're still interested:
http://download.systemimager.org/~arighi/linux/patches/io-throttle/

There's no consistent changes respect to the latest version I posted to the LKML, just rebasing to the recent kernels.

Quoted 9 lines

Some quick thoughts about this approach though.

- It is not a proportional weight controller. It is more of limiting bandwidth in absolute numbers for each cgroup on each disk.

  So each cgroup will define a rule for each disk in the system mentioning at what maximum rate that cgroup can issue IO to that disk and throttle the IO from that cgroup if rate has excedded.

Correct. Add also the proportional weight control has been in the TODO list since the early versions, but I never dedicated too much effort to implement this feature, I can focus on this and try to write something if we all think it is worth to be done.

Quoted 8 lines

  Above requirement can create configuration problems.

      - If there are large number of disks in system, per cgroup one shall have to create rules for each disk. Until and unless admin knows what applications are in which cgroup and strictly what disk these applications do IO to and create rules for only those disks.

I don't think this is a huge problem anyway. IMHO a userspace tool, e.g. a script, would be able to efficiently create/modify rules parsing user defined rules in some human-readable form (config files, etc.), even in presence of hundreds of disk. The same is valid for dm-ioband I think.

Quoted 4 lines

      - I think problem gets compounded if there is a hierarchy of logical devices. I think in that case one shall have to create rules for logical devices and not actual physical devices.

With logical devices you mean device-mapper devices (i.e. LVM, software RAID, etc.)? or do you mean that we need to introduce the concept of "logical device" to easily (quickly) configure IO requirements and then map those logical devices to the actual physical devices? In this case I think this can be addressed in userspace. Or maybe I'm totally missing the point here.

Quoted 5 lines

- Because it is not proportional weight distribution, if some cgroup is not using its planned BW, other group sharing the disk can not make use of spare BW.

Right.

Quoted 9 lines

- I think one should know in advance the throughput rate of underlying media and also know competing applications so that one can statically define the BW assigned to each cgroup on each disk.

  This will be difficult. Effective BW extracted out of a rotational media is dependent on the seek pattern so one shall have to either try to make some conservative estimates and try to divide BW (we will not utilize disk fully) or take some peak numbers and divide BW (cgroup might not get the maximum rate configured).

Quoted 6 lines

Correct. I think the proportional weight approach is the only solution to efficiently use the whole BW. OTOH absolute limiting rules offer a better control over QoS, because you can totally remove performance bursts/peaks that could break QoS requirements for short periods of time. So, my "ideal" IO controller should allow to define both rules:
absolute and proportional limits.

I completely agree with Andrea here. The final solution has to have both absolute limits and proportions. But instead of adding a token based approach on top of proportional based system, I have been thinking about modifying the proportional approach to support absolute limits. This might not work, but I think this is an interesting idea to think about.

Here are my thoughts on it so far. We start with the patches that Vivek has sent, and change the notion of weights to percent. That is, the user space specifies percents of disk times instead of weights. We do not put entities in idle tree; whenever an entity is not backlogged, we still keep them in the active trees, and allocate them time slices. But since they have not requests in them, no requests will get dispatched during the time slices allocated to these entities. Moreover, if the percents of all entities do not add upto hundred, we introduce a dummy entity at each level to soak up the rest of "percent". This dummy entity would get time slices just like any other entity, but will not dispatch any requests. With these modification, we can limit entities to their allocated percent of disk time.

We might want a situation in which we want to allow certain entities to exceed their "percent" of disk time, while others should be limited. In this case, we can extend the above mentioned approach by introducing a secondary active tree. All entities which are allowed to exceed their "percent" can be queued in the secondary tree, besides the primary tree. Whenever an idle (or dummy) entity gets a time slice, instead of idling the disk, an entity can be picked from the secondary tree.

The advantage of an approach like this is that it will be relatively smaller modification of the proposed proportional approach. Moreover, entities will be throttled by getting time slices less frequently, instead of being allowed to send a burst and then getting starved (like in ticket based approach). The downside is that this approach sounds unconventional and probably have not been tried in other domains either. Thoughts? opinions?

I will create patches based on the above idea in a few weeks.

Quoted 6 lines

I still have to look closely at your patchset anyway. I will do and give a feedback.

-Andrea

#19 lore raw

Re: [PATCH 01/10] Documentation

In reply to Andrea Righi
Quoted 15 lines · Andrea Righi
Quoted 2 lines · Vivek Goyal
Quoted 1 line · Andrew Morton

On Wed, 11 Mar 2009 21:56:46 -0400 Vivek Goyal <[email protected]> wrote:

[snip]

Quoted 10 lines
Quoted 7 lines

Also.. there are so many IO controller implementations that I've lost track of who is doing what. I do have one private report here that Andreas's controller "is incredibly productive for us and has allowed us to put twice as many users per server with faster times for all users". Which is pretty stunning, although it should be viewed as a condemnation of the current code, I'm afraid.

I had looked briefly at Andrea's implementation in the past. I will look again. I had thought that this approach did not get much traction.

Quoted 7 lines

Hi Vivek, sorry for my late reply. I periodically upload the latest versions of io-throttle here if you're still interested:
http://download.systemimager.org/~arighi/linux/patches/io-throttle/

There's no consistent changes respect to the latest version I posted to the LKML, just rebasing to the recent kernels.

Thanks Andrea. I will spend more time in looking through your patches and do a bit of testing.

Quoted 6 lines
Quoted 6 lines

Some quick thoughts about this approach though.

- It is not a proportional weight controller. It is more of limiting bandwidth in absolute numbers for each cgroup on each disk.

Quoted 9 lines
Quoted 3 lines

So each cgroup will define a rule for each disk in the system mentioning at what maximum rate that cgroup can issue IO to that disk and throttle the IO from that cgroup if rate has excedded.

Correct. Add also the proportional weight control has been in the TODO list since the early versions, but I never dedicated too much effort to implement this feature, I can focus on this and try to write something if we all think it is worth to be done.

Please do have a look at this patchset and would you do it differently to implement proportional weight control?

Few thoughts/queries.

- Max bandwidth control and Prportional weight control are two entirely different ways of controlling the IO. Former one tries to put an upper limit on the IO rate and later one kind of tries to gurantee minmum percentage share of disk.

How does an determine what throughput rate you will get from a disk? That is so much dependent on workload and miscalculations can lead to getting lower BW for a particular cgroup?

I am assuming that one can probably do some random read-write IO test to try to get some idea of disk throughput. If that's the case, then in proportional weight control also you should be able to predict the minimum BW a cgroup will be getting? The only difference will be that a cgroup can get higher BW also if there is no contention present and I am wondring that how getting more BW than promised minumum is harmful?

- I can think of atleast one usage of uppper limit controller where we might have spare IO resources still we don't want to give it to a cgroup because customer has not paid for that kind of service level. In those cases we need to implement uppper limit also.

May be prportional weight and max bw controller can co-exist depending on what user's requirements are.

If yes, then can't this control be done at the same layer/level where proportional weight control is being done? IOW, this set of patches is trying to do prportional weight control at IO scheduler level. I think we should be able to store another max rate as another feature in cgroup (apart from weight) and not dispatch requests from the queue if we have exceeded the max BW as specified by the user?

- Have you thought of doing hierarchical control?

- What happens to the notion of CFQ task classes and task priority. Looks like max bw rule supercede everything. There is no way that an RT task get unlimited amount of disk BW even if it wants to? (There is no notion of RT cgroup etc)

Quoted 19 lines
Quoted 8 lines

Above requirement can create configuration problems.

- If there are large number of disks in system, per cgroup one shall have to create rules for each disk. Until and unless admin knows what applications are in which cgroup and strictly what disk these applications do IO to and create rules for only those disks.

I don't think this is a huge problem anyway. IMHO a userspace tool, e.g. a script, would be able to efficiently create/modify rules parsing user defined rules in some human-readable form (config files, etc.), even in presence of hundreds of disk. The same is valid for dm-ioband I think.

Quoted 4 lines

- I think problem gets compounded if there is a hierarchy of logical devices. I think in that case one shall have to create rules for logical devices and not actual physical devices.

Quoted 6 lines

With logical devices you mean device-mapper devices (i.e. LVM, software RAID, etc.)? or do you mean that we need to introduce the concept of "logical device" to easily (quickly) configure IO requirements and then map those logical devices to the actual physical devices? In this case I think this can be addressed in userspace. Or maybe I'm totally missing the point here.

Yes, I meant LVM, Software RAID etc. So if I have got many disks in the system and I have created software raid on some of them, I need to create rules for lvm devices or physical devices behind those lvm devices? I am assuming that it will be logical devices.

So I need to know exactly to what all devices applications in a particular cgroup is going to do IO, and also know exactly how many cgroups are contending for that cgroup, and also know what worst case disk rate I can expect from that device and then I can do a good job of giving a reasonable value to the max rate of that cgroup on a particular device?

Quoted 19 lines
Quoted 6 lines

- Because it is not proportional weight distribution, if some cgroup is not using its planned BW, other group sharing the disk can not make use of spare BW.

Right.

Quoted 9 lines

- I think one should know in advance the throughput rate of underlying media and also know competing applications so that one can statically define the BW assigned to each cgroup on each disk.

This will be difficult. Effective BW extracted out of a rotational media is dependent on the seek pattern so one shall have to either try to make some conservative estimates and try to divide BW (we will not utilize disk fully) or take some peak numbers and divide BW (cgroup might not get the maximum rate configured).

Quoted 5 lines

Correct. I think the proportional weight approach is the only solution to efficiently use the whole BW. OTOH absolute limiting rules offer a better control over QoS, because you can totally remove performance bursts/peaks that could break QoS requirements for short periods of time.

Can you please give little more details here regarding how QoS requirements are not met with proportional weight?

Quoted 5 lines

So, my "ideal" IO controller should allow to define both rules:
absolute and proportional limits.

I still have to look closely at your patchset anyway. I will do and give a feedback.

You feedback is always welcome.

Thanks
Vivek

A message that is not in the archive (sent off-list or not yet received): [email protected]

#21 lore raw

Re: [PATCH 01/10] Documentation

Quoted 46 lines · Vivek Goyal
Quoted 15 lines · Andrea Righi
Quoted 2 lines · Vivek Goyal
Quoted 1 line · Andrew Morton

On Wed, 11 Mar 2009 21:56:46 -0400 Vivek Goyal <[email protected]> wrote:

[snip]

Quoted 10 lines
Quoted 7 lines

Also.. there are so many IO controller implementations that I've lost track of who is doing what. I do have one private report here that Andreas's controller "is incredibly productive for us and has allowed us to put twice as many users per server with faster times for all users". Which is pretty stunning, although it should be viewed as a condemnation of the current code, I'm afraid.

I had looked briefly at Andrea's implementation in the past. I will look again. I had thought that this approach did not get much traction.

Quoted 7 lines

Hi Vivek, sorry for my late reply. I periodically upload the latest versions of io-throttle here if you're still interested:
http://download.systemimager.org/~arighi/linux/patches/io-throttle/

There's no consistent changes respect to the latest version I posted to the LKML, just rebasing to the recent kernels.

Thanks Andrea. I will spend more time in looking through your patches and do a bit of testing.

Quoted 6 lines
Quoted 6 lines

Some quick thoughts about this approach though.

- It is not a proportional weight controller. It is more of limiting bandwidth in absolute numbers for each cgroup on each disk.

Quoted 9 lines
Quoted 3 lines

So each cgroup will define a rule for each disk in the system mentioning at what maximum rate that cgroup can issue IO to that disk and throttle the IO from that cgroup if rate has excedded.

Correct. Add also the proportional weight control has been in the TODO list since the early versions, but I never dedicated too much effort to implement this feature, I can focus on this and try to write something if we all think it is worth to be done.

Please do have a look at this patchset and would you do it differently to implement proportional weight control?

Quoted 6 lines

Few thoughts/queries.

- Max bandwidth control and Prportional weight control are two entirely different ways of controlling the IO. Former one tries to put an upper limit on the IO rate and later one kind of tries to gurantee minmum percentage share of disk.

Agree.

Quoted 11 lines

How does an determine what throughput rate you will get from a disk? That is so much dependent on workload and miscalculations can lead to getting lower BW for a particular cgroup?

I am assuming that one can probably do some random read-write IO test to try to get some idea of disk throughput. If that's the case, then in proportional weight control also you should be able to predict the minimum BW a cgroup will be getting? The only difference will be that a cgroup can get higher BW also if there is no contention present and I am wondring that how getting more BW than promised minumum is harmful?

IMHO we shouldn't care too much on how to extract the exact BW from a disk. With proportional weights we can directly map different levels of service to different weights.

With absolute limiting we can measure the consumed BW post facto and try to do the best to satisfy the limits defined by the user (absolute max, min or proportional). Predict a priori how much BW will consume a particular application's workload is a very hard task (maybe even impossible) and probably it doesn't give huge advantages respect to the approach we're currently using. I think this is true for both solutions.

Quoted 9 lines

- I can think of atleast one usage of uppper limit controller where we might have spare IO resources still we don't want to give it to a cgroup because customer has not paid for that kind of service level. In those cases we need to implement uppper limit also.

May be prportional weight and max bw controller can co-exist depending on what user's requirements are.

Quoted 6 lines

If yes, then can't this control be done at the same layer/level where proportional weight control is being done? IOW, this set of patches is trying to do prportional weight control at IO scheduler level. I think we should be able to store another max rate as another feature in cgroup (apart from weight) and not dispatch requests from the queue if we have exceeded the max BW as specified by the user?

The more I think about a "perfect" solution (at least for my requirements), the more I'm convinced that we need both functionalities.

I think it would be possible to implement both proportional and limiting rules at the same level (e.g., the IO scheduler), but we need also to address the memory consumption problem (I still need to review your patchset in details and I'm going to test it soon :), so I don't know if you already addressed this issue).

IOW if we simply don't dispatch requests and we don't throttle the tasks in the cgroup that exceeds its limit, how do we avoid the waste of memory due to the succeeding IO requests and the increasingly dirty pages in the page cache (that are also hard to reclaim)? I may be wrong, but I think we talked about this problem in a previous email... sorry I don't find the discussion in my mail archives.

IMHO a nice approach would be to measure IO consumption at the IO scheduler level, and control IO applying proportional weights / absolute limits _both_ at the IO scheduler / elevator level _and_ at the same time block the tasks from dirtying memory that will generate additional IO requests.

Anyway, there's no need to provide this with a single IO controller, we could split the problem in two parts: 1) provide a proportional / absolute IO controller in the IO schedulers and 2) allow to set, for example, a maximum limit of dirty pages for each cgroup.

Maybe I'm just repeating what we already said in a previous discussion... in this case sorry for the duplicate thoughts. :)

Quoted 3 lines

- Have you thought of doing hierarchical control?

Providing hiearchies in cgroups is in general expensive, deeper hierarchies imply checking all the way up to the root cgroup, so I think we need to be very careful and be aware of the trade-offs before providing such feature. For this particular case (IO controller) wouldn't it be simpler and more efficient to just ignore hierarchies in the kernel and opportunely handle them in userspace? for absolute limiting rules this isn't difficult at all, just imagine a config file and a script or a deamon that dynamically create the opportune cgroups and configure them accordingly to what is defined in the configuration file.

I think we can simply define hierarchical dependencies in the configuration file, translate them in absolute values and use the absolute values to configure the cgroups' properties.

For example, we can just check that the BW allocated for a particular parent cgroup is not greater than the total BW allocated for the children. And for each child just use the min(parent_BW, BW) or equally divide the parent's BW among the children, etc.

Quoted 4 lines

- What happens to the notion of CFQ task classes and task priority. Looks like max bw rule supercede everything. There is no way that an RT task get unlimited amount of disk BW even if it wants to? (There is no notion of RT cgroup etc)

What about moving all the RT tasks in a separate cgroup with unlimited BW?

Quoted 32 lines
Quoted 20 lines
Quoted 9 lines

Above requirement can create configuration problems.

- If there are large number of disks in system, per cgroup one shall have to create rules for each disk. Until and unless admin knows what applications are in which cgroup and strictly what disk these applications do IO to and create rules for only those disks.

I don't think this is a huge problem anyway. IMHO a userspace tool, e.g. a script, would be able to efficiently create/modify rules parsing user defined rules in some human-readable form (config files, etc.), even in presence of hundreds of disk. The same is valid for dm-ioband I think.

Quoted 4 lines

- I think problem gets compounded if there is a hierarchy of logical devices. I think in that case one shall have to create rules for logical devices and not actual physical devices.

Quoted 6 lines

With logical devices you mean device-mapper devices (i.e. LVM, software RAID, etc.)? or do you mean that we need to introduce the concept of "logical device" to easily (quickly) configure IO requirements and then map those logical devices to the actual physical devices? In this case I think this can be addressed in userspace. Or maybe I'm totally missing the point here.

Yes, I meant LVM, Software RAID etc. So if I have got many disks in the system and I have created software raid on some of them, I need to create rules for lvm devices or physical devices behind those lvm devices? I am assuming that it will be logical devices.

Quoted 5 lines

So I need to know exactly to what all devices applications in a particular cgroup is going to do IO, and also know exactly how many cgroups are contending for that cgroup, and also know what worst case disk rate I can expect from that device and then I can do a good job of giving a reasonable value to the max rate of that cgroup on a particular device?

ok, I understand. For these cases dm-ioband perfectly addresses the problem. For the general case, I think the only solution is to provide a common interface that each dm subsystem must call to account IO and apply limiting and proportional rules.

Quoted 20 lines
Quoted 20 lines
Quoted 7 lines

- Because it is not proportional weight distribution, if some cgroup is not using its planned BW, other group sharing the disk can not make use of spare BW.

Right.

Quoted 9 lines

- I think one should know in advance the throughput rate of underlying media and also know competing applications so that one can statically define the BW assigned to each cgroup on each disk.

This will be difficult. Effective BW extracted out of a rotational media is dependent on the seek pattern so one shall have to either try to make some conservative estimates and try to divide BW (we will not utilize disk fully) or take some peak numbers and divide BW (cgroup might not get the maximum rate configured).

Quoted 8 lines
Quoted 5 lines

Correct. I think the proportional weight approach is the only solution to efficiently use the whole BW. OTOH absolute limiting rules offer a better control over QoS, because you can totally remove performance bursts/peaks that could break QoS requirements for short periods of time.

Can you please give little more details here regarding how QoS requirements are not met with proportional weight?

With proportional weights the whole bandwidth is allocated if no one else is using it. When IO is submitted other tasks with a higher weight can be forced to sleep until the IO generated by the low weight tasks is not completely dispatched. Or any extent of the priority inversion problems.

Maybe it's not an issue at all for the most part of the cases, but using a solution that is able to provide also a real partitioning of the available resources can be profitely used by those who need to guarantee _strict_ BW requirements (soft real-time, maximize the responsiveness of certain services, etc.), because in this case we're sure that a certain amount of "spare" BW will be always available when needed by some "critical" services.

Quoted 11 lines
Quoted 6 lines

So, my "ideal" IO controller should allow to define both rules:
absolute and proportional limits.

I still have to look closely at your patchset anyway. I will do and give a feedback.

You feedback is always welcome.

Thanks
Vivek

Thanks,
-Andrea

#22 lore raw

Re: [PATCH 01/10] Documentation

In reply to Vivek Goyal
Quoted 46 lines · Vivek Goyal
Quoted 15 lines · Andrea Righi
Quoted 2 lines · Vivek Goyal
Quoted 1 line · Andrew Morton

On Wed, 11 Mar 2009 21:56:46 -0400 Vivek Goyal <[email protected]> wrote:

[snip]

Quoted 10 lines
Quoted 7 lines

Also.. there are so many IO controller implementations that I've lost track of who is doing what. I do have one private report here that Andreas's controller "is incredibly productive for us and has allowed us to put twice as many users per server with faster times for all users". Which is pretty stunning, although it should be viewed as a condemnation of the current code, I'm afraid.

I had looked briefly at Andrea's implementation in the past. I will look again. I had thought that this approach did not get much traction.

Quoted 7 lines

Hi Vivek, sorry for my late reply. I periodically upload the latest versions of io-throttle here if you're still interested:
http://download.systemimager.org/~arighi/linux/patches/io-throttle/

There's no consistent changes respect to the latest version I posted to the LKML, just rebasing to the recent kernels.

Thanks Andrea. I will spend more time in looking through your patches and do a bit of testing.

Quoted 6 lines
Quoted 6 lines

Some quick thoughts about this approach though.

- It is not a proportional weight controller. It is more of limiting bandwidth in absolute numbers for each cgroup on each disk.

Quoted 9 lines
Quoted 3 lines

So each cgroup will define a rule for each disk in the system mentioning at what maximum rate that cgroup can issue IO to that disk and throttle the IO from that cgroup if rate has excedded.

Correct. Add also the proportional weight control has been in the TODO list since the early versions, but I never dedicated too much effort to implement this feature, I can focus on this and try to write something if we all think it is worth to be done.

Please do have a look at this patchset and would you do it differently to implement proportional weight control?

Quoted 6 lines

Few thoughts/queries.

- Max bandwidth control and Prportional weight control are two entirely different ways of controlling the IO. Former one tries to put an upper limit on the IO rate and later one kind of tries to gurantee minmum percentage share of disk.

Agree.

Quoted 11 lines

How does an determine what throughput rate you will get from a disk? That is so much dependent on workload and miscalculations can lead to getting lower BW for a particular cgroup?

I am assuming that one can probably do some random read-write IO test to try to get some idea of disk throughput. If that's the case, then in proportional weight control also you should be able to predict the minimum BW a cgroup will be getting? The only difference will be that a cgroup can get higher BW also if there is no contention present and I am wondring that how getting more BW than promised minumum is harmful?

IMHO we shouldn't care too much on how to extract the exact BW from a disk. With proportional weights we can directly map different levels of service to different weights.

With absolute limiting we can measure the consumed BW post facto and try to do the best to satisfy the limits defined by the user (absolute max, min or proportional). Predict a priori how much BW will consume a particular application's workload is a very hard task (maybe even impossible) and probably it doesn't give huge advantages respect to the approach we're currently using. I think this is true for both solutions.

Quoted 9 lines

- I can think of atleast one usage of uppper limit controller where we might have spare IO resources still we don't want to give it to a cgroup because customer has not paid for that kind of service level. In those cases we need to implement uppper limit also.

May be prportional weight and max bw controller can co-exist depending on what user's requirements are.

Quoted 6 lines

If yes, then can't this control be done at the same layer/level where proportional weight control is being done? IOW, this set of patches is trying to do prportional weight control at IO scheduler level. I think we should be able to store another max rate as another feature in cgroup (apart from weight) and not dispatch requests from the queue if we have exceeded the max BW as specified by the user?

The more I think about a "perfect" solution (at least for my requirements), the more I'm convinced that we need both functionalities.

I think it would be possible to implement both proportional and limiting rules at the same level (e.g., the IO scheduler), but we need also to address the memory consumption problem (I still need to review your patchset in details and I'm going to test it soon :), so I don't know if you already addressed this issue).

IOW if we simply don't dispatch requests and we don't throttle the tasks in the cgroup that exceeds its limit, how do we avoid the waste of memory due to the succeeding IO requests and the increasingly dirty pages in the page cache (that are also hard to reclaim)? I may be wrong, but I think we talked about this problem in a previous email... sorry I don't find the discussion in my mail archives.

IMHO a nice approach would be to measure IO consumption at the IO scheduler level, and control IO applying proportional weights / absolute limits _both_ at the IO scheduler / elevator level _and_ at the same time block the tasks from dirtying memory that will generate additional IO requests.

Anyway, there's no need to provide this with a single IO controller, we could split the problem in two parts: 1) provide a proportional / absolute IO controller in the IO schedulers and 2) allow to set, for example, a maximum limit of dirty pages for each cgroup.

Maybe I'm just repeating what we already said in a previous discussion... in this case sorry for the duplicate thoughts. :)

Quoted 3 lines

- Have you thought of doing hierarchical control?

Providing hiearchies in cgroups is in general expensive, deeper hierarchies imply checking all the way up to the root cgroup, so I think we need to be very careful and be aware of the trade-offs before providing such feature. For this particular case (IO controller) wouldn't it be simpler and more efficient to just ignore hierarchies in the kernel and opportunely handle them in userspace? for absolute limiting rules this isn't difficult at all, just imagine a config file and a script or a deamon that dynamically create the opportune cgroups and configure them accordingly to what is defined in the configuration file.

I think we can simply define hierarchical dependencies in the configuration file, translate them in absolute values and use the absolute values to configure the cgroups' properties.

For example, we can just check that the BW allocated for a particular parent cgroup is not greater than the total BW allocated for the children. And for each child just use the min(parent_BW, BW) or equally divide the parent's BW among the children, etc.

Quoted 4 lines

- What happens to the notion of CFQ task classes and task priority. Looks like max bw rule supercede everything. There is no way that an RT task get unlimited amount of disk BW even if it wants to? (There is no notion of RT cgroup etc)

What about moving all the RT tasks in a separate cgroup with unlimited BW?

Quoted 32 lines
Quoted 20 lines
Quoted 9 lines

Above requirement can create configuration problems.

- If there are large number of disks in system, per cgroup one shall have to create rules for each disk. Until and unless admin knows what applications are in which cgroup and strictly what disk these applications do IO to and create rules for only those disks.

I don't think this is a huge problem anyway. IMHO a userspace tool, e.g. a script, would be able to efficiently create/modify rules parsing user defined rules in some human-readable form (config files, etc.), even in presence of hundreds of disk. The same is valid for dm-ioband I think.

Quoted 4 lines

- I think problem gets compounded if there is a hierarchy of logical devices. I think in that case one shall have to create rules for logical devices and not actual physical devices.

Quoted 6 lines

With logical devices you mean device-mapper devices (i.e. LVM, software RAID, etc.)? or do you mean that we need to introduce the concept of "logical device" to easily (quickly) configure IO requirements and then map those logical devices to the actual physical devices? In this case I think this can be addressed in userspace. Or maybe I'm totally missing the point here.

Yes, I meant LVM, Software RAID etc. So if I have got many disks in the system and I have created software raid on some of them, I need to create rules for lvm devices or physical devices behind those lvm devices? I am assuming that it will be logical devices.

Quoted 5 lines

So I need to know exactly to what all devices applications in a particular cgroup is going to do IO, and also know exactly how many cgroups are contending for that cgroup, and also know what worst case disk rate I can expect from that device and then I can do a good job of giving a reasonable value to the max rate of that cgroup on a particular device?

ok, I understand. For these cases dm-ioband perfectly addresses the problem. For the general case, I think the only solution is to provide a common interface that each dm subsystem must call to account IO and apply limiting and proportional rules.

Quoted 20 lines
Quoted 20 lines
Quoted 7 lines

- Because it is not proportional weight distribution, if some cgroup is not using its planned BW, other group sharing the disk can not make use of spare BW.

Right.

Quoted 9 lines

- I think one should know in advance the throughput rate of underlying media and also know competing applications so that one can statically define the BW assigned to each cgroup on each disk.

This will be difficult. Effective BW extracted out of a rotational media is dependent on the seek pattern so one shall have to either try to make some conservative estimates and try to divide BW (we will not utilize disk fully) or take some peak numbers and divide BW (cgroup might not get the maximum rate configured).

Quoted 8 lines
Quoted 5 lines

Correct. I think the proportional weight approach is the only solution to efficiently use the whole BW. OTOH absolute limiting rules offer a better control over QoS, because you can totally remove performance bursts/peaks that could break QoS requirements for short periods of time.

Can you please give little more details here regarding how QoS requirements are not met with proportional weight?

With proportional weights the whole bandwidth is allocated if no one else is using it. When IO is submitted other tasks with a higher weight can be forced to sleep until the IO generated by the low weight tasks is not completely dispatched. Or any extent of the priority inversion problems.

Maybe it's not an issue at all for the most part of the cases, but using a solution that is able to provide also a real partitioning of the available resources can be profitely used by those who need to guarantee _strict_ BW requirements (soft real-time, maximize the responsiveness of certain services, etc.), because in this case we're sure that a certain amount of "spare" BW will be always available when needed by some "critical" services.

Quoted 11 lines
Quoted 6 lines

So, my "ideal" IO controller should allow to define both rules:
absolute and proportional limits.

I still have to look closely at your patchset anyway. I will do and give a feedback.

You feedback is always welcome.

Thanks
Vivek

Thanks,
-Andrea

#23 lore raw

Re: [PATCH 01/10] Documentation

On Wed, Apr 08, 2009 at 10:37:59PM +0200, Andrea Righi wrote:

[..]

Quoted 16 lines
Quoted 9 lines

- I can think of atleast one usage of uppper limit controller where we might have spare IO resources still we don't want to give it to a cgroup because customer has not paid for that kind of service level. In those cases we need to implement uppper limit also.

May be prportional weight and max bw controller can co-exist depending on what user's requirements are.

Quoted 6 lines

If yes, then can't this control be done at the same layer/level where proportional weight control is being done? IOW, this set of patches is trying to do prportional weight control at IO scheduler level. I think we should be able to store another max rate as another feature in cgroup (apart from weight) and not dispatch requests from the queue if we have exceeded the max BW as specified by the user?

Quoted 3 lines

The more I think about a "perfect" solution (at least for my requirements), the more I'm convinced that we need both functionalities.

I agree here. In some scenarios people might want to put an upper cap on BW even if more BW is available and in some scenarios people will like to do proportional distribution and let one get more share of disk if it is free.

Quoted 6 lines

I think it would be possible to implement both proportional and limiting rules at the same level (e.g., the IO scheduler), but we need also to address the memory consumption problem (I still need to review your patchset in details and I'm going to test it soon :), so I don't know if you already addressed this issue).

Can you please elaborate a bit on this? Are you concerned about that data structures created to solve the problem consume a lot of memory?

Quoted 13 lines

IOW if we simply don't dispatch requests and we don't throttle the tasks in the cgroup that exceeds its limit, how do we avoid the waste of memory due to the succeeding IO requests and the increasingly dirty pages in the page cache (that are also hard to reclaim)? I may be wrong, but I think we talked about this problem in a previous email... sorry I don't find the discussion in my mail archives.

IMHO a nice approach would be to measure IO consumption at the IO scheduler level, and control IO applying proportional weights / absolute limits _both_ at the IO scheduler / elevator level _and_ at the same time block the tasks from dirtying memory that will generate additional IO requests.

Quoted 5 lines

Anyway, there's no need to provide this with a single IO controller, we could split the problem in two parts: 1) provide a proportional / absolute IO controller in the IO schedulers and 2) allow to set, for example, a maximum limit of dirty pages for each cgroup.

I think setting a maximum limit on dirty pages is an interesting thought. It sounds like as if memory controller can handle it?

I guess currently memory controller puts limit on total amount of memory consumed by cgroup and there are no knobs on type of memory consumed. So if one can limit amount of dirty page cache memory per cgroup, it automatically throttles the aysnc writes at the input itself.

So I agree that if we can limit the process from dirtying too much of memory than IO scheduler level controller should be able to do both proportional weight and max bw controller.

Currently doing proportional weight control for async writes is very tricky. I am not seeing constantly backlogged traffic at IO scheudler level and hence two different weight processes seem to be getting same BW.

I will dive deeper into the patches on dm-ioband to see how they have solved this issue. Looks like they are just waiting longer for slowest group to consume its tokens and that will keep the disk idle. Extended delays might now show up immediately as performance hog, because it might also promote increased merging but it should lead to increased latency of response. And proving latency issues is hard. :-)

Quoted 18 lines

Maybe I'm just repeating what we already said in a previous discussion... in this case sorry for the duplicate thoughts. :)

Quoted 3 lines

- Have you thought of doing hierarchical control?

Providing hiearchies in cgroups is in general expensive, deeper hierarchies imply checking all the way up to the root cgroup, so I think we need to be very careful and be aware of the trade-offs before providing such feature. For this particular case (IO controller) wouldn't it be simpler and more efficient to just ignore hierarchies in the kernel and opportunely handle them in userspace? for absolute limiting rules this isn't difficult at all, just imagine a config file and a script or a deamon that dynamically create the opportune cgroups and configure them accordingly to what is defined in the configuration file.

Quoted 8 lines

I think we can simply define hierarchical dependencies in the configuration file, translate them in absolute values and use the absolute values to configure the cgroups' properties.

For example, we can just check that the BW allocated for a particular parent cgroup is not greater than the total BW allocated for the children. And for each child just use the min(parent_BW, BW) or equally divide the parent's BW among the children, etc.

IIUC, you are saying that allow hiearchy in user space and then flatten it out and pass it to kernel?

Hmm.., agree that handling hierarchies is hard and expensive. But at the same time rest of the controllers like cpu and memory are handling it in kernel so it probably makes sense to keep the IO controller also in line.

In practice I am not expecting deep hiearchices. May be 2- 3 levels would be good for most of the people.

Quoted 8 lines
Quoted 5 lines

- What happens to the notion of CFQ task classes and task priority. Looks like max bw rule supercede everything. There is no way that an RT task get unlimited amount of disk BW even if it wants to? (There is no notion of RT cgroup etc)

What about moving all the RT tasks in a separate cgroup with unlimited BW?

Hmm.., I think that should work. I have yet to look at your patches in detail but it looks like unlimited BW group will not be throttled at all hence RT tasks can just go right through without getting impacted.

Quoted 70 lines
Quoted 33 lines
Quoted 21 lines
Quoted 10 lines

Above requirement can create configuration problems.

- If there are large number of disks in system, per cgroup one shall have to create rules for each disk. Until and unless admin knows what applications are in which cgroup and strictly what disk these applications do IO to and create rules for only those disks.

I don't think this is a huge problem anyway. IMHO a userspace tool, e.g. a script, would be able to efficiently create/modify rules parsing user defined rules in some human-readable form (config files, etc.), even in presence of hundreds of disk. The same is valid for dm-ioband I think.

Quoted 4 lines

- I think problem gets compounded if there is a hierarchy of logical devices. I think in that case one shall have to create rules for logical devices and not actual physical devices.

Quoted 6 lines

With logical devices you mean device-mapper devices (i.e. LVM, software RAID, etc.)? or do you mean that we need to introduce the concept of "logical device" to easily (quickly) configure IO requirements and then map those logical devices to the actual physical devices? In this case I think this can be addressed in userspace. Or maybe I'm totally missing the point here.

Yes, I meant LVM, Software RAID etc. So if I have got many disks in the system and I have created software raid on some of them, I need to create rules for lvm devices or physical devices behind those lvm devices? I am assuming that it will be logical devices.

Quoted 5 lines

So I need to know exactly to what all devices applications in a particular cgroup is going to do IO, and also know exactly how many cgroups are contending for that cgroup, and also know what worst case disk rate I can expect from that device and then I can do a good job of giving a reasonable value to the max rate of that cgroup on a particular device?

ok, I understand. For these cases dm-ioband perfectly addresses the problem. For the general case, I think the only solution is to provide a common interface that each dm subsystem must call to account IO and apply limiting and proportional rules.

Quoted 26 lines
Quoted 25 lines
Quoted 7 lines

- Because it is not proportional weight distribution, if some cgroup is not using its planned BW, other group sharing the disk can not make use of spare BW.

Right.

Quoted 9 lines

- I think one should know in advance the throughput rate of underlying media and also know competing applications so that one can statically define the BW assigned to each cgroup on each disk.

This will be difficult. Effective BW extracted out of a rotational media is dependent on the seek pattern so one shall have to either try to make some conservative estimates and try to divide BW (we will not utilize disk fully) or take some peak numbers and divide BW (cgroup might not get the maximum rate configured).

Correct. I think the proportional weight approach is the only solution to efficiently use the whole BW. OTOH absolute limiting rules offer a better control over QoS, because you can totally remove performance bursts/peaks that could break QoS requirements for short periods of time.

Quoted 8 lines
Quoted 2 lines

Can you please give little more details here regarding how QoS requirements are not met with proportional weight?

With proportional weights the whole bandwidth is allocated if no one else is using it. When IO is submitted other tasks with a higher weight can be forced to sleep until the IO generated by the low weight tasks is not completely dispatched. Or any extent of the priority inversion problems.

Hmm..., I am not very sure here. When admin is allocating the weights, he has the whole picture. He knows how many groups are conteding for the disk and what could be the worst case scenario. So if I have got two groups with A and B with weight 1 and 2 and both are contending, then as an admin one would expect to get 33% of BW for group A in worst case (if group B is continuously backlogged). If B is not contending than A can get 100% of BW. So while configuring the system, will one not plan for worst case (33% for A, and 66 % for B)?

Quoted 9 lines

Maybe it's not an issue at all for the most part of the cases, but using a solution that is able to provide also a real partitioning of the available resources can be profitely used by those who need to guarantee _strict_ BW requirements (soft real-time, maximize the responsiveness of certain services, etc.), because in this case we're sure that a certain amount of "spare" BW will be always available when needed by some "critical" services.

Will the same thing not happen in proportional weight? If it is an RT application, one can put it in RT groups to make sure it always gets the BW first even if there is contention.

Even in regular group, the moment you issue the IO and IO scheduler sees it, you will start getting your reserved share according to your weight.

How it will be different in the case of io throttling? Even if I don't utilize the disk fully, cfq will still put the new guy in the queue and then try to give its share (based on prio).

Are you saying that by keeping disk relatively free, the latency of response for soft real time application will become better? In that case can't one simply underprovision the disk?

But having said that I am not disputing the need of max BW controller as some people have expressed the need of a constant BW view and don't want too big a fluctuations even if BW is available. Max BW controller can't gurantee the minumum BW hence can't avoid the fluctuations completely, but it can still help in smoothing the traffic because other competitiors will be stopped from doing too much of IO.

Thanks
Vivek

Quoted 14 lines
Quoted 11 lines
Quoted 6 lines

So, my "ideal" IO controller should allow to define both rules:
absolute and proportional limits.

I still have to look closely at your patchset anyway. I will do and give a feedback.

You feedback is always welcome.

Thanks
Vivek

Thanks,
-Andrea

A message that is not in the archive (sent off-list or not yet received): [email protected]

#25 lore raw

Re: [PATCH 01/10] Documentation

Quoted 22 lines · Vivek Goyal

On Wed, Apr 08, 2009 at 10:37:59PM +0200, Andrea Righi wrote:

[..]

Quoted 19 lines
Quoted 9 lines

- I can think of atleast one usage of uppper limit controller where we might have spare IO resources still we don't want to give it to a cgroup because customer has not paid for that kind of service level. In those cases we need to implement uppper limit also.

May be prportional weight and max bw controller can co-exist depending on what user's requirements are.

Quoted 6 lines

If yes, then can't this control be done at the same layer/level where proportional weight control is being done? IOW, this set of patches is trying to do prportional weight control at IO scheduler level. I think we should be able to store another max rate as another feature in cgroup (apart from weight) and not dispatch requests from the queue if we have exceeded the max BW as specified by the user?

The more I think about a "perfect" solution (at least for my requirements), the more I'm convinced that we need both functionalities.

hard limits vs work conserving argument again :). I agree, we need both of the functionalities. I think first the aim should be to get the proportional weight functionality and then look at doing hard limits.

[..]

Quoted 27 lines
Quoted 15 lines
Quoted 3 lines

- Have you thought of doing hierarchical control?

Providing hiearchies in cgroups is in general expensive, deeper hierarchies imply checking all the way up to the root cgroup, so I think we need to be very careful and be aware of the trade-offs before providing such feature. For this particular case (IO controller) wouldn't it be simpler and more efficient to just ignore hierarchies in the kernel and opportunely handle them in userspace? for absolute limiting rules this isn't difficult at all, just imagine a config file and a script or a deamon that dynamically create the opportune cgroups and configure them accordingly to what is defined in the configuration file.

Quoted 8 lines

I think we can simply define hierarchical dependencies in the configuration file, translate them in absolute values and use the absolute values to configure the cgroups' properties.

For example, we can just check that the BW allocated for a particular parent cgroup is not greater than the total BW allocated for the children. And for each child just use the min(parent_BW, BW) or equally divide the parent's BW among the children, etc.

IIUC, you are saying that allow hiearchy in user space and then flatten it out and pass it to kernel?

Quoted 7 lines

Hmm.., agree that handling hierarchies is hard and expensive. But at the same time rest of the controllers like cpu and memory are handling it in kernel so it probably makes sense to keep the IO controller also in line.

In practice I am not expecting deep hiearchices. May be 2- 3 levels would be good for most of the people.

FWIW, even in the CPU controller having deep hierarchies is not a good idea. I think this can be documented for IO Controller as well. Beyond that, we realized that having a proportional system and doing it in userspace is not a good idea. It would require a lot of calculations dependending on the system load. (Because, the sub-group should be just the same as a process in the parent group). Having hierarchy in the kernel just makes it way more easier and way more accurate.

Quoted 6 lines
Quoted 6 lines
Quoted 5 lines

- What happens to the notion of CFQ task classes and task priority. Looks like max bw rule supercede everything. There is no way that an RT task get unlimited amount of disk BW even if it wants to? (There is no notion of RT cgroup etc)

Quoted 7 lines
Quoted 2 lines

What about moving all the RT tasks in a separate cgroup with unlimited BW?

Hmm.., I think that should work. I have yet to look at your patches in detail but it looks like unlimited BW group will not be throttled at all hence RT tasks can just go right through without getting impacted.

This is where the cpu scheduler design helped a lot :). Having different classes for differnet types of processes allowed us to handle them separately.

thanks,

Signature · 2 lines
regards,
Dhaval
#26 lore raw

Re: [PATCH 01/10] Documentation

In reply to a message not in the archive
Quoted 9 lines · Vivek Goyal
Quoted 6 lines

I think it would be possible to implement both proportional and limiting rules at the same level (e.g., the IO scheduler), but we need also to address the memory consumption problem (I still need to review your patchset in details and I'm going to test it soon :), so I don't know if you already addressed this issue).

Can you please elaborate a bit on this? Are you concerned about that data structures created to solve the problem consume a lot of memory?

Sorry I was not very clear here. With memory consumption I mean wasting the memory with hard/slow reclaimable dirty pages or pending IO requests.

If there's only a global limit on dirty pages, any cgroup can exhaust that limit and cause other cgroups/processes to block when they try to write to disk.

But, ok, the IO controller is not probably the best place to implement such functionality. I should rework on the per cgroup dirty_ratio:

https://lists.linux-foundation.org/pipermail/containers/2008-September/013140.html

Last time we focused too much on the best interfaces to define dirty pages limit, and I never re-posted an updated version of this patchset. Now I think we can simply provide the same dirty_ratio/dirty_bytes interface that we provide globally, but per cgroup.

Quoted 14 lines
Quoted 14 lines

IOW if we simply don't dispatch requests and we don't throttle the tasks in the cgroup that exceeds its limit, how do we avoid the waste of memory due to the succeeding IO requests and the increasingly dirty pages in the page cache (that are also hard to reclaim)? I may be wrong, but I think we talked about this problem in a previous email... sorry I don't find the discussion in my mail archives.

IMHO a nice approach would be to measure IO consumption at the IO scheduler level, and control IO applying proportional weights / absolute limits _both_ at the IO scheduler / elevator level _and_ at the same time block the tasks from dirtying memory that will generate additional IO requests.

Quoted 8 lines
Quoted 5 lines

Anyway, there's no need to provide this with a single IO controller, we could split the problem in two parts: 1) provide a proportional / absolute IO controller in the IO schedulers and 2) allow to set, for example, a maximum limit of dirty pages for each cgroup.

I think setting a maximum limit on dirty pages is an interesting thought. It sounds like as if memory controller can handle it?

Exactly, the same above.

Quoted 65 lines

I guess currently memory controller puts limit on total amount of memory consumed by cgroup and there are no knobs on type of memory consumed. So if one can limit amount of dirty page cache memory per cgroup, it automatically throttles the aysnc writes at the input itself.

So I agree that if we can limit the process from dirtying too much of memory than IO scheduler level controller should be able to do both proportional weight and max bw controller.

Currently doing proportional weight control for async writes is very tricky. I am not seeing constantly backlogged traffic at IO scheudler level and hence two different weight processes seem to be getting same BW.

I will dive deeper into the patches on dm-ioband to see how they have solved this issue. Looks like they are just waiting longer for slowest group to consume its tokens and that will keep the disk idle. Extended delays might now show up immediately as performance hog, because it might also promote increased merging but it should lead to increased latency of response. And proving latency issues is hard. :-)

Quoted 18 lines

Maybe I'm just repeating what we already said in a previous discussion... in this case sorry for the duplicate thoughts. :)

Quoted 3 lines

- Have you thought of doing hierarchical control?

Providing hiearchies in cgroups is in general expensive, deeper hierarchies imply checking all the way up to the root cgroup, so I think we need to be very careful and be aware of the trade-offs before providing such feature. For this particular case (IO controller) wouldn't it be simpler and more efficient to just ignore hierarchies in the kernel and opportunely handle them in userspace? for absolute limiting rules this isn't difficult at all, just imagine a config file and a script or a deamon that dynamically create the opportune cgroups and configure them accordingly to what is defined in the configuration file.

Quoted 8 lines

I think we can simply define hierarchical dependencies in the configuration file, translate them in absolute values and use the absolute values to configure the cgroups' properties.

For example, we can just check that the BW allocated for a particular parent cgroup is not greater than the total BW allocated for the children. And for each child just use the min(parent_BW, BW) or equally divide the parent's BW among the children, etc.

IIUC, you are saying that allow hiearchy in user space and then flatten it out and pass it to kernel?

Hmm.., agree that handling hierarchies is hard and expensive. But at the same time rest of the controllers like cpu and memory are handling it in kernel so it probably makes sense to keep the IO controller also in line.

In practice I am not expecting deep hiearchices. May be 2- 3 levels would be good for most of the people.

Quoted 6 lines
Quoted 5 lines

- What happens to the notion of CFQ task classes and task priority. Looks like max bw rule supercede everything. There is no way that an RT task get unlimited amount of disk BW even if it wants to? (There is no notion of RT cgroup etc)

Quoted 6 lines
Quoted 2 lines

What about moving all the RT tasks in a separate cgroup with unlimited BW?

Hmm.., I think that should work. I have yet to look at your patches in detail but it looks like unlimited BW group will not be throttled at all hence RT tasks can just go right through without getting impacted.

Correct.

Quoted 80 lines
Quoted 71 lines
Quoted 34 lines
Quoted 22 lines
Quoted 11 lines

Above requirement can create configuration problems.

- If there are large number of disks in system, per cgroup one shall have to create rules for each disk. Until and unless admin knows what applications are in which cgroup and strictly what disk these applications do IO to and create rules for only those disks.

I don't think this is a huge problem anyway. IMHO a userspace tool, e.g. a script, would be able to efficiently create/modify rules parsing user defined rules in some human-readable form (config files, etc.), even in presence of hundreds of disk. The same is valid for dm-ioband I think.

Quoted 4 lines

- I think problem gets compounded if there is a hierarchy of logical devices. I think in that case one shall have to create rules for logical devices and not actual physical devices.

Quoted 6 lines

With logical devices you mean device-mapper devices (i.e. LVM, software RAID, etc.)? or do you mean that we need to introduce the concept of "logical device" to easily (quickly) configure IO requirements and then map those logical devices to the actual physical devices? In this case I think this can be addressed in userspace. Or maybe I'm totally missing the point here.

Yes, I meant LVM, Software RAID etc. So if I have got many disks in the system and I have created software raid on some of them, I need to create rules for lvm devices or physical devices behind those lvm devices? I am assuming that it will be logical devices.

Quoted 5 lines

So I need to know exactly to what all devices applications in a particular cgroup is going to do IO, and also know exactly how many cgroups are contending for that cgroup, and also know what worst case disk rate I can expect from that device and then I can do a good job of giving a reasonable value to the max rate of that cgroup on a particular device?

ok, I understand. For these cases dm-ioband perfectly addresses the problem. For the general case, I think the only solution is to provide a common interface that each dm subsystem must call to account IO and apply limiting and proportional rules.

Quoted 26 lines
Quoted 25 lines
Quoted 7 lines

- Because it is not proportional weight distribution, if some cgroup is not using its planned BW, other group sharing the disk can not make use of spare BW.

Right.

Quoted 9 lines

- I think one should know in advance the throughput rate of underlying media and also know competing applications so that one can statically define the BW assigned to each cgroup on each disk.

This will be difficult. Effective BW extracted out of a rotational media is dependent on the seek pattern so one shall have to either try to make some conservative estimates and try to divide BW (we will not utilize disk fully) or take some peak numbers and divide BW (cgroup might not get the maximum rate configured).

Correct. I think the proportional weight approach is the only solution to efficiently use the whole BW. OTOH absolute limiting rules offer a better control over QoS, because you can totally remove performance bursts/peaks that could break QoS requirements for short periods of time.

Quoted 8 lines
Quoted 2 lines

Can you please give little more details here regarding how QoS requirements are not met with proportional weight?

With proportional weights the whole bandwidth is allocated if no one else is using it. When IO is submitted other tasks with a higher weight can be forced to sleep until the IO generated by the low weight tasks is not completely dispatched. Or any extent of the priority inversion problems.

Quoted 8 lines

Hmm..., I am not very sure here. When admin is allocating the weights, he has the whole picture. He knows how many groups are conteding for the disk and what could be the worst case scenario. So if I have got two groups with A and B with weight 1 and 2 and both are contending, then as an admin one would expect to get 33% of BW for group A in worst case (if group B is continuously backlogged). If B is not contending than A can get 100% of BW. So while configuring the system, will one not plan for worst case (33% for A, and 66 % for B)?

OK, I'm quite convinced.. :)

To a large degree, if we want to provide a BW reservation strategy we must provide an interface that allows cgroups to ask for time slices such as max/min 5 IO requests every 50ms or something like that. Probably the same functionality can be achieved translating time slices from weights, percentages or absolute BW limits.

Quoted 26 lines
Quoted 10 lines

Maybe it's not an issue at all for the most part of the cases, but using a solution that is able to provide also a real partitioning of the available resources can be profitely used by those who need to guarantee _strict_ BW requirements (soft real-time, maximize the responsiveness of certain services, etc.), because in this case we're sure that a certain amount of "spare" BW will be always available when needed by some "critical" services.

Will the same thing not happen in proportional weight? If it is an RT application, one can put it in RT groups to make sure it always gets the BW first even if there is contention.

Even in regular group, the moment you issue the IO and IO scheduler sees it, you will start getting your reserved share according to your weight.

How it will be different in the case of io throttling? Even if I don't utilize the disk fully, cfq will still put the new guy in the queue and then try to give its share (based on prio).

Are you saying that by keeping disk relatively free, the latency of response for soft real time application will become better? In that case can't one simply underprovision the disk?

Quoted 6 lines

But having said that I am not disputing the need of max BW controller as some people have expressed the need of a constant BW view and don't want too big a fluctuations even if BW is available. Max BW controller can't gurantee the minumum BW hence can't avoid the fluctuations completely, but it can still help in smoothing the traffic because other competitiors will be stopped from doing too much of IO.

Agree.

-Andrea

#27 lore raw

Re: [PATCH 01/10] Documentation

In reply to Vivek Goyal
Quoted 22 lines · Vivek Goyal

On Wed, Apr 08, 2009 at 10:37:59PM +0200, Andrea Righi wrote:

[..]

Quoted 19 lines
Quoted 9 lines

- I can think of atleast one usage of uppper limit controller where we might have spare IO resources still we don't want to give it to a cgroup because customer has not paid for that kind of service level. In those cases we need to implement uppper limit also.

May be prportional weight and max bw controller can co-exist depending on what user's requirements are.

Quoted 6 lines

If yes, then can't this control be done at the same layer/level where proportional weight control is being done? IOW, this set of patches is trying to do prportional weight control at IO scheduler level. I think we should be able to store another max rate as another feature in cgroup (apart from weight) and not dispatch requests from the queue if we have exceeded the max BW as specified by the user?

The more I think about a "perfect" solution (at least for my requirements), the more I'm convinced that we need both functionalities.

hard limits vs work conserving argument again :). I agree, we need both of the functionalities. I think first the aim should be to get the proportional weight functionality and then look at doing hard limits.

[..]

Quoted 27 lines
Quoted 15 lines
Quoted 3 lines

- Have you thought of doing hierarchical control?

Providing hiearchies in cgroups is in general expensive, deeper hierarchies imply checking all the way up to the root cgroup, so I think we need to be very careful and be aware of the trade-offs before providing such feature. For this particular case (IO controller) wouldn't it be simpler and more efficient to just ignore hierarchies in the kernel and opportunely handle them in userspace? for absolute limiting rules this isn't difficult at all, just imagine a config file and a script or a deamon that dynamically create the opportune cgroups and configure them accordingly to what is defined in the configuration file.

Quoted 8 lines

I think we can simply define hierarchical dependencies in the configuration file, translate them in absolute values and use the absolute values to configure the cgroups' properties.

For example, we can just check that the BW allocated for a particular parent cgroup is not greater than the total BW allocated for the children. And for each child just use the min(parent_BW, BW) or equally divide the parent's BW among the children, etc.

IIUC, you are saying that allow hiearchy in user space and then flatten it out and pass it to kernel?

Quoted 7 lines

Hmm.., agree that handling hierarchies is hard and expensive. But at the same time rest of the controllers like cpu and memory are handling it in kernel so it probably makes sense to keep the IO controller also in line.

In practice I am not expecting deep hiearchices. May be 2- 3 levels would be good for most of the people.

FWIW, even in the CPU controller having deep hierarchies is not a good idea. I think this can be documented for IO Controller as well. Beyond that, we realized that having a proportional system and doing it in userspace is not a good idea. It would require a lot of calculations dependending on the system load. (Because, the sub-group should be just the same as a process in the parent group). Having hierarchy in the kernel just makes it way more easier and way more accurate.

Quoted 6 lines
Quoted 6 lines
Quoted 5 lines

- What happens to the notion of CFQ task classes and task priority. Looks like max bw rule supercede everything. There is no way that an RT task get unlimited amount of disk BW even if it wants to? (There is no notion of RT cgroup etc)

Quoted 7 lines
Quoted 2 lines

What about moving all the RT tasks in a separate cgroup with unlimited BW?

Hmm.., I think that should work. I have yet to look at your patches in detail but it looks like unlimited BW group will not be throttled at all hence RT tasks can just go right through without getting impacted.

This is where the cpu scheduler design helped a lot :). Having different classes for differnet types of processes allowed us to handle them separately.

thanks,

Signature · 2 lines
regards,
Dhaval
#28 lore raw

IO Controller discussion (Was: Re: [PATCH 01/10] Documentation)

Quoted 20 lines · Dhaval Giani
Quoted 19 lines · Vivek Goyal

On Wed, Apr 08, 2009 at 10:37:59PM +0200, Andrea Righi wrote:

[..]

Quoted 16 lines
Quoted 15 lines

- I can think of atleast one usage of uppper limit controller where we might have spare IO resources still we don't want to give it to a cgroup because customer has not paid for that kind of service level. In those cases we need to implement uppper limit also.

May be prportional weight and max bw controller can co-exist depending on what user's requirements are.

If yes, then can't this control be done at the same layer/level where proportional weight control is being done? IOW, this set of patches is trying to do prportional weight control at IO scheduler level. I think we should be able to store another max rate as another feature in cgroup (apart from weight) and not dispatch requests from the queue if we have exceeded the max BW as specified by the user?

Quoted 8 lines
Quoted 3 lines
Quoted 3 lines

The more I think about a "perfect" solution (at least for my requirements), the more I'm convinced that we need both functionalities.

hard limits vs work conserving argument again :). I agree, we need both of the functionalities. I think first the aim should be to get the proportional weight functionality and then look at doing hard limits.

Agreed.

Quoted 37 lines

[..]

Quoted 27 lines
Quoted 15 lines
Quoted 3 lines

- Have you thought of doing hierarchical control?

Providing hiearchies in cgroups is in general expensive, deeper hierarchies imply checking all the way up to the root cgroup, so I think we need to be very careful and be aware of the trade-offs before providing such feature. For this particular case (IO controller) wouldn't it be simpler and more efficient to just ignore hierarchies in the kernel and opportunely handle them in userspace? for absolute limiting rules this isn't difficult at all, just imagine a config file and a script or a deamon that dynamically create the opportune cgroups and configure them accordingly to what is defined in the configuration file.

Quoted 8 lines

I think we can simply define hierarchical dependencies in the configuration file, translate them in absolute values and use the absolute values to configure the cgroups' properties.

For example, we can just check that the BW allocated for a particular parent cgroup is not greater than the total BW allocated for the children. And for each child just use the min(parent_BW, BW) or equally divide the parent's BW among the children, etc.

IIUC, you are saying that allow hiearchy in user space and then flatten it out and pass it to kernel?

Quoted 7 lines

Hmm.., agree that handling hierarchies is hard and expensive. But at the same time rest of the controllers like cpu and memory are handling it in kernel so it probably makes sense to keep the IO controller also in line.

In practice I am not expecting deep hiearchices. May be 2- 3 levels would be good for most of the people.

Quoted 7 lines

FWIW, even in the CPU controller having deep hierarchies is not a good idea. I think this can be documented for IO Controller as well. Beyond that, we realized that having a proportional system and doing it in userspace is not a good idea. It would require a lot of calculations dependending on the system load. (Because, the sub-group should be just the same as a process in the parent group). Having hierarchy in the kernel just makes it way more easier and way more accurate.

Agreed. I will prefer to keep hierarchical support in kernel inline with other controllers.

Quoted 10 lines
Quoted 10 lines
Quoted 9 lines
Quoted 6 lines

- What happens to the notion of CFQ task classes and task priority. Looks like max bw rule supercede everything. There is no way that an RT task get unlimited amount of disk BW even if it wants to? (There is no notion of RT cgroup etc)

What about moving all the RT tasks in a separate cgroup with unlimited BW?

Quoted 8 lines
Quoted 4 lines

Hmm.., I think that should work. I have yet to look at your patches in detail but it looks like unlimited BW group will not be throttled at all hence RT tasks can just go right through without getting impacted.

This is where the cpu scheduler design helped a lot :). Having different classes for differnet types of processes allowed us to handle them separately.

In common layer scheduling approach, we do have separate classes (RT, BE and IDLE) and scheduling is done accordingly. Code primarily taken fro bfq and cfq.

dm-ioband has no notion of separate classes and everything was being treated at same level which is a problem as end level IO scheduler will loose its capability to differentiate we mixup he things above it.

Time to play with max bw controller patches and then I can probably have more insights into it.

Thanks
Vivek

#29 lore raw

Re: [PATCH 01/10] Documentation

In reply to Vivek Goyal
Quoted 9 lines · Vivek Goyal
Quoted 6 lines

I think it would be possible to implement both proportional and limiting rules at the same level (e.g., the IO scheduler), but we need also to address the memory consumption problem (I still need to review your patchset in details and I'm going to test it soon :), so I don't know if you already addressed this issue).

Can you please elaborate a bit on this? Are you concerned about that data structures created to solve the problem consume a lot of memory?

Sorry I was not very clear here. With memory consumption I mean wasting the memory with hard/slow reclaimable dirty pages or pending IO requests.

If there's only a global limit on dirty pages, any cgroup can exhaust that limit and cause other cgroups/processes to block when they try to write to disk.

But, ok, the IO controller is not probably the best place to implement such functionality. I should rework on the per cgroup dirty_ratio:

https://lists.linux-foundation.org/pipermail/containers/2008-September/013140.html

Last time we focused too much on the best interfaces to define dirty pages limit, and I never re-posted an updated version of this patchset. Now I think we can simply provide the same dirty_ratio/dirty_bytes interface that we provide globally, but per cgroup.

Quoted 14 lines
Quoted 14 lines

IOW if we simply don't dispatch requests and we don't throttle the tasks in the cgroup that exceeds its limit, how do we avoid the waste of memory due to the succeeding IO requests and the increasingly dirty pages in the page cache (that are also hard to reclaim)? I may be wrong, but I think we talked about this problem in a previous email... sorry I don't find the discussion in my mail archives.

IMHO a nice approach would be to measure IO consumption at the IO scheduler level, and control IO applying proportional weights / absolute limits _both_ at the IO scheduler / elevator level _and_ at the same time block the tasks from dirtying memory that will generate additional IO requests.

Quoted 8 lines
Quoted 5 lines

Anyway, there's no need to provide this with a single IO controller, we could split the problem in two parts: 1) provide a proportional / absolute IO controller in the IO schedulers and 2) allow to set, for example, a maximum limit of dirty pages for each cgroup.

I think setting a maximum limit on dirty pages is an interesting thought. It sounds like as if memory controller can handle it?

Exactly, the same above.

Quoted 65 lines

I guess currently memory controller puts limit on total amount of memory consumed by cgroup and there are no knobs on type of memory consumed. So if one can limit amount of dirty page cache memory per cgroup, it automatically throttles the aysnc writes at the input itself.

So I agree that if we can limit the process from dirtying too much of memory than IO scheduler level controller should be able to do both proportional weight and max bw controller.

Currently doing proportional weight control for async writes is very tricky. I am not seeing constantly backlogged traffic at IO scheudler level and hence two different weight processes seem to be getting same BW.

I will dive deeper into the patches on dm-ioband to see how they have solved this issue. Looks like they are just waiting longer for slowest group to consume its tokens and that will keep the disk idle. Extended delays might now show up immediately as performance hog, because it might also promote increased merging but it should lead to increased latency of response. And proving latency issues is hard. :-)

Quoted 18 lines

Maybe I'm just repeating what we already said in a previous discussion... in this case sorry for the duplicate thoughts. :)

Quoted 3 lines

- Have you thought of doing hierarchical control?

Providing hiearchies in cgroups is in general expensive, deeper hierarchies imply checking all the way up to the root cgroup, so I think we need to be very careful and be aware of the trade-offs before providing such feature. For this particular case (IO controller) wouldn't it be simpler and more efficient to just ignore hierarchies in the kernel and opportunely handle them in userspace? for absolute limiting rules this isn't difficult at all, just imagine a config file and a script or a deamon that dynamically create the opportune cgroups and configure them accordingly to what is defined in the configuration file.

Quoted 8 lines

I think we can simply define hierarchical dependencies in the configuration file, translate them in absolute values and use the absolute values to configure the cgroups' properties.

For example, we can just check that the BW allocated for a particular parent cgroup is not greater than the total BW allocated for the children. And for each child just use the min(parent_BW, BW) or equally divide the parent's BW among the children, etc.

IIUC, you are saying that allow hiearchy in user space and then flatten it out and pass it to kernel?

Hmm.., agree that handling hierarchies is hard and expensive. But at the same time rest of the controllers like cpu and memory are handling it in kernel so it probably makes sense to keep the IO controller also in line.

In practice I am not expecting deep hiearchices. May be 2- 3 levels would be good for most of the people.

Quoted 6 lines
Quoted 5 lines

- What happens to the notion of CFQ task classes and task priority. Looks like max bw rule supercede everything. There is no way that an RT task get unlimited amount of disk BW even if it wants to? (There is no notion of RT cgroup etc)

Quoted 6 lines
Quoted 2 lines

What about moving all the RT tasks in a separate cgroup with unlimited BW?

Hmm.., I think that should work. I have yet to look at your patches in detail but it looks like unlimited BW group will not be throttled at all hence RT tasks can just go right through without getting impacted.

Correct.

Quoted 80 lines
Quoted 71 lines
Quoted 34 lines
Quoted 22 lines
Quoted 11 lines

Above requirement can create configuration problems.

- If there are large number of disks in system, per cgroup one shall have to create rules for each disk. Until and unless admin knows what applications are in which cgroup and strictly what disk these applications do IO to and create rules for only those disks.

I don't think this is a huge problem anyway. IMHO a userspace tool, e.g. a script, would be able to efficiently create/modify rules parsing user defined rules in some human-readable form (config files, etc.), even in presence of hundreds of disk. The same is valid for dm-ioband I think.

Quoted 4 lines

- I think problem gets compounded if there is a hierarchy of logical devices. I think in that case one shall have to create rules for logical devices and not actual physical devices.

Quoted 6 lines

With logical devices you mean device-mapper devices (i.e. LVM, software RAID, etc.)? or do you mean that we need to introduce the concept of "logical device" to easily (quickly) configure IO requirements and then map those logical devices to the actual physical devices? In this case I think this can be addressed in userspace. Or maybe I'm totally missing the point here.

Yes, I meant LVM, Software RAID etc. So if I have got many disks in the system and I have created software raid on some of them, I need to create rules for lvm devices or physical devices behind those lvm devices? I am assuming that it will be logical devices.

Quoted 5 lines

So I need to know exactly to what all devices applications in a particular cgroup is going to do IO, and also know exactly how many cgroups are contending for that cgroup, and also know what worst case disk rate I can expect from that device and then I can do a good job of giving a reasonable value to the max rate of that cgroup on a particular device?

ok, I understand. For these cases dm-ioband perfectly addresses the problem. For the general case, I think the only solution is to provide a common interface that each dm subsystem must call to account IO and apply limiting and proportional rules.

Quoted 26 lines
Quoted 25 lines
Quoted 7 lines

- Because it is not proportional weight distribution, if some cgroup is not using its planned BW, other group sharing the disk can not make use of spare BW.

Right.

Quoted 9 lines

- I think one should know in advance the throughput rate of underlying media and also know competing applications so that one can statically define the BW assigned to each cgroup on each disk.

This will be difficult. Effective BW extracted out of a rotational media is dependent on the seek pattern so one shall have to either try to make some conservative estimates and try to divide BW (we will not utilize disk fully) or take some peak numbers and divide BW (cgroup might not get the maximum rate configured).

Correct. I think the proportional weight approach is the only solution to efficiently use the whole BW. OTOH absolute limiting rules offer a better control over QoS, because you can totally remove performance bursts/peaks that could break QoS requirements for short periods of time.

Quoted 8 lines
Quoted 2 lines

Can you please give little more details here regarding how QoS requirements are not met with proportional weight?

With proportional weights the whole bandwidth is allocated if no one else is using it. When IO is submitted other tasks with a higher weight can be forced to sleep until the IO generated by the low weight tasks is not completely dispatched. Or any extent of the priority inversion problems.

Quoted 8 lines

Hmm..., I am not very sure here. When admin is allocating the weights, he has the whole picture. He knows how many groups are conteding for the disk and what could be the worst case scenario. So if I have got two groups with A and B with weight 1 and 2 and both are contending, then as an admin one would expect to get 33% of BW for group A in worst case (if group B is continuously backlogged). If B is not contending than A can get 100% of BW. So while configuring the system, will one not plan for worst case (33% for A, and 66 % for B)?

OK, I'm quite convinced.. :)

To a large degree, if we want to provide a BW reservation strategy we must provide an interface that allows cgroups to ask for time slices such as max/min 5 IO requests every 50ms or something like that. Probably the same functionality can be achieved translating time slices from weights, percentages or absolute BW limits.

Quoted 26 lines
Quoted 10 lines

Maybe it's not an issue at all for the most part of the cases, but using a solution that is able to provide also a real partitioning of the available resources can be profitely used by those who need to guarantee _strict_ BW requirements (soft real-time, maximize the responsiveness of certain services, etc.), because in this case we're sure that a certain amount of "spare" BW will be always available when needed by some "critical" services.

Will the same thing not happen in proportional weight? If it is an RT application, one can put it in RT groups to make sure it always gets the BW first even if there is contention.

Even in regular group, the moment you issue the IO and IO scheduler sees it, you will start getting your reserved share according to your weight.

How it will be different in the case of io throttling? Even if I don't utilize the disk fully, cfq will still put the new guy in the queue and then try to give its share (based on prio).

Are you saying that by keeping disk relatively free, the latency of response for soft real time application will become better? In that case can't one simply underprovision the disk?

Quoted 6 lines

But having said that I am not disputing the need of max BW controller as some people have expressed the need of a constant BW view and don't want too big a fluctuations even if BW is available. Max BW controller can't gurantee the minumum BW hence can't avoid the fluctuations completely, but it can still help in smoothing the traffic because other competitiors will be stopped from doing too much of IO.

Agree.

-Andrea

#30 lore raw

IO controller discussion (Was: Re: [PATCH 01/10] Documentation)

Quoted 49 lines · Andrea Righi
Quoted 9 lines · Vivek Goyal
Quoted 6 lines

I think it would be possible to implement both proportional and limiting rules at the same level (e.g., the IO scheduler), but we need also to address the memory consumption problem (I still need to review your patchset in details and I'm going to test it soon :), so I don't know if you already addressed this issue).

Can you please elaborate a bit on this? Are you concerned about that data structures created to solve the problem consume a lot of memory?

Sorry I was not very clear here. With memory consumption I mean wasting the memory with hard/slow reclaimable dirty pages or pending IO requests.

If there's only a global limit on dirty pages, any cgroup can exhaust that limit and cause other cgroups/processes to block when they try to write to disk.

But, ok, the IO controller is not probably the best place to implement such functionality. I should rework on the per cgroup dirty_ratio:

https://lists.linux-foundation.org/pipermail/containers/2008-September/013140.html

Last time we focused too much on the best interfaces to define dirty pages limit, and I never re-posted an updated version of this patchset. Now I think we can simply provide the same dirty_ratio/dirty_bytes interface that we provide globally, but per cgroup.

Quoted 20 lines
Quoted 19 lines

IOW if we simply don't dispatch requests and we don't throttle the tasks in the cgroup that exceeds its limit, how do we avoid the waste of memory due to the succeeding IO requests and the increasingly dirty pages in the page cache (that are also hard to reclaim)? I may be wrong, but I think we talked about this problem in a previous email... sorry I don't find the discussion in my mail archives.

IMHO a nice approach would be to measure IO consumption at the IO scheduler level, and control IO applying proportional weights / absolute limits _both_ at the IO scheduler / elevator level _and_ at the same time block the tasks from dirtying memory that will generate additional IO requests.

Anyway, there's no need to provide this with a single IO controller, we could split the problem in two parts: 1) provide a proportional / absolute IO controller in the IO schedulers and 2) allow to set, for example, a maximum limit of dirty pages for each cgroup.

Quoted 4 lines
Quoted 2 lines

I think setting a maximum limit on dirty pages is an interesting thought. It sounds like as if memory controller can handle it?

Exactly, the same above.

Thinking more about it. Memory controller can probably enforce the higher limit but it would not easily translate into a fixed upper async write rate. Till the process hits the page cache limit or is slowed down by dirty page writeout, it can get a very high async write BW.

So memory controller page cache limit will help but it would not direclty translate into what max bw limit patches are doing.

Even if we do max bw control at IO scheduler level, async writes are problematic again. IO controller will not be able to throttle the process until it sees actuall write request. In big memory systems, writeout might not happen for some time and till then it will see a high throughput.

So doing async write throttling at higher layer and not at IO scheduler layer gives us the opprotunity to produce more accurate results.

For sync requests, I think IO scheduler max bw control should work fine.

BTW, andrea, what is the use case of your patches? Andrew had mentioned that some people are already using it. I am curious to know will a proportional BW controller will solve the issues/requirements of these people or they have specific requirement of traffic shaping and max bw controller only.

[..]

Quoted 18 lines
Quoted 9 lines
Quoted 8 lines
Quoted 2 lines

Can you please give little more details here regarding how QoS requirements are not met with proportional weight?

With proportional weights the whole bandwidth is allocated if no one else is using it. When IO is submitted other tasks with a higher weight can be forced to sleep until the IO generated by the low weight tasks is not completely dispatched. Or any extent of the priority inversion problems.

Quoted 8 lines

Hmm..., I am not very sure here. When admin is allocating the weights, he has the whole picture. He knows how many groups are conteding for the disk and what could be the worst case scenario. So if I have got two groups with A and B with weight 1 and 2 and both are contending, then as an admin one would expect to get 33% of BW for group A in worst case (if group B is continuously backlogged). If B is not contending than A can get 100% of BW. So while configuring the system, will one not plan for worst case (33% for A, and 66 % for B)?

Quoted 7 lines

OK, I'm quite convinced.. :)

To a large degree, if we want to provide a BW reservation strategy we must provide an interface that allows cgroups to ask for time slices such as max/min 5 IO requests every 50ms or something like that. Probably the same functionality can be achieved translating time slices from weights, percentages or absolute BW limits.

Ok, I would like to split it in two parts.

I think providng minimum gurantee in absolute terms like 5 IO request every 50ms will be very hard because IO scheduler has no control over how many competitors are there. An easier thing will be to have minimum gurantees on share basis. For minimum BW (disk time slice) gurantee, admin shall have to create right cgroup hierarchy and assign weights properly and then admin can calculate what % of disk slice a particular group will get as minimum gurantee. (This is more complicated than this as there are time slices which are not accounted to any groups. During queue switch cfq starts the time slice counting only after first request has completed to offset the impact of seeking and i guess also NCQ).

I think it should be possible to give max bandwidth gurantees in absolute terms, like io/s or sectors/sec or MB/sec etc, because only thing IO scheduler has to do is to not allow dispatch from a particular queue if it has crossed its limit and then either let the disk idle or move onto next eligible queue.

The only issue here will be async writes. max bw gurantee for async writes at IO scheduler level might not mean much to application because of page cache.

Thanks
Vivek

#31 lore raw

IO controller discussion (Was: Re: [PATCH 01/10] Documentation)

In reply to Andrea Righi
Quoted 49 lines · Andrea Righi
Quoted 9 lines · Vivek Goyal
Quoted 6 lines

I think it would be possible to implement both proportional and limiting rules at the same level (e.g., the IO scheduler), but we need also to address the memory consumption problem (I still need to review your patchset in details and I'm going to test it soon :), so I don't know if you already addressed this issue).

Can you please elaborate a bit on this? Are you concerned about that data structures created to solve the problem consume a lot of memory?

Sorry I was not very clear here. With memory consumption I mean wasting the memory with hard/slow reclaimable dirty pages or pending IO requests.

If there's only a global limit on dirty pages, any cgroup can exhaust that limit and cause other cgroups/processes to block when they try to write to disk.

But, ok, the IO controller is not probably the best place to implement such functionality. I should rework on the per cgroup dirty_ratio:

https://lists.linux-foundation.org/pipermail/containers/2008-September/013140.html

Last time we focused too much on the best interfaces to define dirty pages limit, and I never re-posted an updated version of this patchset. Now I think we can simply provide the same dirty_ratio/dirty_bytes interface that we provide globally, but per cgroup.

Quoted 20 lines
Quoted 19 lines

IOW if we simply don't dispatch requests and we don't throttle the tasks in the cgroup that exceeds its limit, how do we avoid the waste of memory due to the succeeding IO requests and the increasingly dirty pages in the page cache (that are also hard to reclaim)? I may be wrong, but I think we talked about this problem in a previous email... sorry I don't find the discussion in my mail archives.

IMHO a nice approach would be to measure IO consumption at the IO scheduler level, and control IO applying proportional weights / absolute limits _both_ at the IO scheduler / elevator level _and_ at the same time block the tasks from dirtying memory that will generate additional IO requests.

Anyway, there's no need to provide this with a single IO controller, we could split the problem in two parts: 1) provide a proportional / absolute IO controller in the IO schedulers and 2) allow to set, for example, a maximum limit of dirty pages for each cgroup.

Quoted 4 lines
Quoted 2 lines

I think setting a maximum limit on dirty pages is an interesting thought. It sounds like as if memory controller can handle it?

Exactly, the same above.

Thinking more about it. Memory controller can probably enforce the higher limit but it would not easily translate into a fixed upper async write rate. Till the process hits the page cache limit or is slowed down by dirty page writeout, it can get a very high async write BW.

So memory controller page cache limit will help but it would not direclty translate into what max bw limit patches are doing.

Even if we do max bw control at IO scheduler level, async writes are problematic again. IO controller will not be able to throttle the process until it sees actuall write request. In big memory systems, writeout might not happen for some time and till then it will see a high throughput.

So doing async write throttling at higher layer and not at IO scheduler layer gives us the opprotunity to produce more accurate results.

For sync requests, I think IO scheduler max bw control should work fine.

BTW, andrea, what is the use case of your patches? Andrew had mentioned that some people are already using it. I am curious to know will a proportional BW controller will solve the issues/requirements of these people or they have specific requirement of traffic shaping and max bw controller only.

[..]

Quoted 18 lines
Quoted 9 lines
Quoted 8 lines
Quoted 2 lines

Can you please give little more details here regarding how QoS requirements are not met with proportional weight?

With proportional weights the whole bandwidth is allocated if no one else is using it. When IO is submitted other tasks with a higher weight can be forced to sleep until the IO generated by the low weight tasks is not completely dispatched. Or any extent of the priority inversion problems.

Quoted 8 lines

Hmm..., I am not very sure here. When admin is allocating the weights, he has the whole picture. He knows how many groups are conteding for the disk and what could be the worst case scenario. So if I have got two groups with A and B with weight 1 and 2 and both are contending, then as an admin one would expect to get 33% of BW for group A in worst case (if group B is continuously backlogged). If B is not contending than A can get 100% of BW. So while configuring the system, will one not plan for worst case (33% for A, and 66 % for B)?

Quoted 7 lines

OK, I'm quite convinced.. :)

To a large degree, if we want to provide a BW reservation strategy we must provide an interface that allows cgroups to ask for time slices such as max/min 5 IO requests every 50ms or something like that. Probably the same functionality can be achieved translating time slices from weights, percentages or absolute BW limits.

Ok, I would like to split it in two parts.

I think providng minimum gurantee in absolute terms like 5 IO request every 50ms will be very hard because IO scheduler has no control over how many competitors are there. An easier thing will be to have minimum gurantees on share basis. For minimum BW (disk time slice) gurantee, admin shall have to create right cgroup hierarchy and assign weights properly and then admin can calculate what % of disk slice a particular group will get as minimum gurantee. (This is more complicated than this as there are time slices which are not accounted to any groups. During queue switch cfq starts the time slice counting only after first request has completed to offset the impact of seeking and i guess also NCQ).

I think it should be possible to give max bandwidth gurantees in absolute terms, like io/s or sectors/sec or MB/sec etc, because only thing IO scheduler has to do is to not allow dispatch from a particular queue if it has crossed its limit and then either let the disk idle or move onto next eligible queue.

The only issue here will be async writes. max bw gurantee for async writes at IO scheduler level might not mean much to application because of page cache.

Thanks
Vivek

A message that is not in the archive (sent off-list or not yet received): [email protected]

#33 lore raw

Re: IO controller discussion (Was: Re: [PATCH 01/10] Documentation)

Quoted 63 lines · Vivek Goyal
Quoted 49 lines · Andrea Righi
Quoted 9 lines · Vivek Goyal
Quoted 6 lines

I think it would be possible to implement both proportional and limiting rules at the same level (e.g., the IO scheduler), but we need also to address the memory consumption problem (I still need to review your patchset in details and I'm going to test it soon :), so I don't know if you already addressed this issue).

Can you please elaborate a bit on this? Are you concerned about that data structures created to solve the problem consume a lot of memory?

Sorry I was not very clear here. With memory consumption I mean wasting the memory with hard/slow reclaimable dirty pages or pending IO requests.

If there's only a global limit on dirty pages, any cgroup can exhaust that limit and cause other cgroups/processes to block when they try to write to disk.

But, ok, the IO controller is not probably the best place to implement such functionality. I should rework on the per cgroup dirty_ratio:

https://lists.linux-foundation.org/pipermail/containers/2008-September/013140.html

Last time we focused too much on the best interfaces to define dirty pages limit, and I never re-posted an updated version of this patchset. Now I think we can simply provide the same dirty_ratio/dirty_bytes interface that we provide globally, but per cgroup.

Quoted 20 lines
Quoted 19 lines

IOW if we simply don't dispatch requests and we don't throttle the tasks in the cgroup that exceeds its limit, how do we avoid the waste of memory due to the succeeding IO requests and the increasingly dirty pages in the page cache (that are also hard to reclaim)? I may be wrong, but I think we talked about this problem in a previous email... sorry I don't find the discussion in my mail archives.

IMHO a nice approach would be to measure IO consumption at the IO scheduler level, and control IO applying proportional weights / absolute limits _both_ at the IO scheduler / elevator level _and_ at the same time block the tasks from dirtying memory that will generate additional IO requests.

Anyway, there's no need to provide this with a single IO controller, we could split the problem in two parts: 1) provide a proportional / absolute IO controller in the IO schedulers and 2) allow to set, for example, a maximum limit of dirty pages for each cgroup.

Quoted 4 lines
Quoted 2 lines

I think setting a maximum limit on dirty pages is an interesting thought. It sounds like as if memory controller can handle it?

Exactly, the same above.

Thinking more about it. Memory controller can probably enforce the higher limit but it would not easily translate into a fixed upper async write rate. Till the process hits the page cache limit or is slowed down by dirty page writeout, it can get a very high async write BW.

So memory controller page cache limit will help but it would not direclty translate into what max bw limit patches are doing.

Quoted 7 lines

Even if we do max bw control at IO scheduler level, async writes are problematic again. IO controller will not be able to throttle the process until it sees actuall write request. In big memory systems, writeout might not happen for some time and till then it will see a high throughput.

So doing async write throttling at higher layer and not at IO scheduler layer gives us the opprotunity to produce more accurate results.

Wouldn't 'doing control on writes at a higher layer' have the same problems as the ones we talk about in dm-ioband? What if the cgroup being throttled for dirtying pages has a high weight assigned to it at the IO scheduler level? What if there are threads of different classes within that cgroup, and we would want to let RT task dirty the pages before BE tasks? I am not sure all these questions make sense, but just wanted to raise issues that might pop up.

If the whole system is designed with cgroups in mind, then throttling at IO scheduler layer should lead to backlog, that could be seen at higher level. For example, if a cgroup is not getting service at IO scheduler level, it should run out of request descriptors, and thus the thread writing back dirty pages should notice it (if its pdflush, blocking it is probably not the best idea). And that should mean the cgroup should hit the dirty threshold, and disallow the task to dirty further pages. There is a possibility though that getting all this right might be an overkill and we can get away with a simpler solution. One possibility seems to be that we provide some feedback from IO scheduling layer to higher layers, that cgroup is hitting its write bandwith limit, and should not be allowed to dirty any more pages.

Quoted 62 lines

For sync requests, I think IO scheduler max bw control should work fine.

BTW, andrea, what is the use case of your patches? Andrew had mentioned that some people are already using it. I am curious to know will a proportional BW controller will solve the issues/requirements of these people or they have specific requirement of traffic shaping and max bw controller only.

[..]

Quoted 18 lines
Quoted 9 lines
Quoted 8 lines
Quoted 2 lines

Can you please give little more details here regarding how QoS requirements are not met with proportional weight?

With proportional weights the whole bandwidth is allocated if no one else is using it. When IO is submitted other tasks with a higher weight can be forced to sleep until the IO generated by the low weight tasks is not completely dispatched. Or any extent of the priority inversion problems.

Quoted 8 lines

Hmm..., I am not very sure here. When admin is allocating the weights, he has the whole picture. He knows how many groups are conteding for the disk and what could be the worst case scenario. So if I have got two groups with A and B with weight 1 and 2 and both are contending, then as an admin one would expect to get 33% of BW for group A in worst case (if group B is continuously backlogged). If B is not contending than A can get 100% of BW. So while configuring the system, will one not plan for worst case (33% for A, and 66 % for B)?

Quoted 7 lines

OK, I'm quite convinced.. :)

To a large degree, if we want to provide a BW reservation strategy we must provide an interface that allows cgroups to ask for time slices such as max/min 5 IO requests every 50ms or something like that. Probably the same functionality can be achieved translating time slices from weights, percentages or absolute BW limits.

Ok, I would like to split it in two parts.

I think providng minimum gurantee in absolute terms like 5 IO request every 50ms will be very hard because IO scheduler has no control over how many competitors are there. An easier thing will be to have minimum gurantees on share basis. For minimum BW (disk time slice) gurantee, admin shall have to create right cgroup hierarchy and assign weights properly and then admin can calculate what % of disk slice a particular group will get as minimum gurantee. (This is more complicated than this as there are time slices which are not accounted to any groups. During queue switch cfq starts the time slice counting only after first request has completed to offset the impact of seeking and i guess also NCQ).

I think it should be possible to give max bandwidth gurantees in absolute terms, like io/s or sectors/sec or MB/sec etc, because only thing IO scheduler has to do is to not allow dispatch from a particular queue if it has crossed its limit and then either let the disk idle or move onto next eligible queue.

The only issue here will be async writes. max bw gurantee for async writes at IO scheduler level might not mean much to application because of page cache.

Thanks
Vivek

#34 lore raw

Re: IO controller discussion (Was: Re: [PATCH 01/10] Documentation)

In reply to a message not in the archive
Quoted 5 lines · Vivek Goyal
Quoted 4 lines
Quoted 2 lines

I think setting a maximum limit on dirty pages is an interesting thought. It sounds like as if memory controller can handle it?

Exactly, the same above.

Quoted 7 lines

Thinking more about it. Memory controller can probably enforce the higher limit but it would not easily translate into a fixed upper async write rate. Till the process hits the page cache limit or is slowed down by dirty page writeout, it can get a very high async write BW.

So memory controller page cache limit will help but it would not direclty translate into what max bw limit patches are doing.

The memory controller can be used to set an upper limit of the dirty pages. When this limit is exceeded the tasks in the cgroup can be forced to write the exceeding dirty pages to disk. At this point the IO controller can: 1) throttle the task that is going to submit the IO requests, if the guy that dirtied the pages was actually the task itself, or 2) delay the submission of those requests to the elevator (or at the IO scheduler level) if it's writeback IO (e.g., made by pdflush).

Both functionalities should allow to have a BW control and avoid that any single cgroup can entirely exhaust the global limit of dirty pages.

Quoted 8 lines

Even if we do max bw control at IO scheduler level, async writes are problematic again. IO controller will not be able to throttle the process until it sees actuall write request. In big memory systems, writeout might not happen for some time and till then it will see a high throughput.

So doing async write throttling at higher layer and not at IO scheduler layer gives us the opprotunity to produce more accurate results.

Totally agree.

Quoted 2 lines

For sync requests, I think IO scheduler max bw control should work fine.

ditto

-Andrea

#35 lore raw

Re: IO controller discussion (Was: Re: [PATCH 01/10] Documentation)

In reply to a message not in the archive
Quoted 125 lines · Vivek Goyal
Quoted 49 lines · Andrea Righi
Quoted 9 lines · Vivek Goyal
Quoted 6 lines

I think it would be possible to implement both proportional and limiting rules at the same level (e.g., the IO scheduler), but we need also to address the memory consumption problem (I still need to review your patchset in details and I'm going to test it soon :), so I don't know if you already addressed this issue).

Can you please elaborate a bit on this? Are you concerned about that data structures created to solve the problem consume a lot of memory?

Sorry I was not very clear here. With memory consumption I mean wasting the memory with hard/slow reclaimable dirty pages or pending IO requests.

If there's only a global limit on dirty pages, any cgroup can exhaust that limit and cause other cgroups/processes to block when they try to write to disk.

But, ok, the IO controller is not probably the best place to implement such functionality. I should rework on the per cgroup dirty_ratio:

https://lists.linux-foundation.org/pipermail/containers/2008-September/013140.html

Last time we focused too much on the best interfaces to define dirty pages limit, and I never re-posted an updated version of this patchset. Now I think we can simply provide the same dirty_ratio/dirty_bytes interface that we provide globally, but per cgroup.

Quoted 20 lines
Quoted 19 lines

IOW if we simply don't dispatch requests and we don't throttle the tasks in the cgroup that exceeds its limit, how do we avoid the waste of memory due to the succeeding IO requests and the increasingly dirty pages in the page cache (that are also hard to reclaim)? I may be wrong, but I think we talked about this problem in a previous email... sorry I don't find the discussion in my mail archives.

IMHO a nice approach would be to measure IO consumption at the IO scheduler level, and control IO applying proportional weights / absolute limits _both_ at the IO scheduler / elevator level _and_ at the same time block the tasks from dirtying memory that will generate additional IO requests.

Anyway, there's no need to provide this with a single IO controller, we could split the problem in two parts: 1) provide a proportional / absolute IO controller in the IO schedulers and 2) allow to set, for example, a maximum limit of dirty pages for each cgroup.

Quoted 4 lines
Quoted 2 lines

I think setting a maximum limit on dirty pages is an interesting thought. It sounds like as if memory controller can handle it?

Exactly, the same above.

Thinking more about it. Memory controller can probably enforce the higher limit but it would not easily translate into a fixed upper async write rate. Till the process hits the page cache limit or is slowed down by dirty page writeout, it can get a very high async write BW.

So memory controller page cache limit will help but it would not direclty translate into what max bw limit patches are doing.

Even if we do max bw control at IO scheduler level, async writes are problematic again. IO controller will not be able to throttle the process until it sees actuall write request. In big memory systems, writeout might not happen for some time and till then it will see a high throughput.

So doing async write throttling at higher layer and not at IO scheduler layer gives us the opprotunity to produce more accurate results.

For sync requests, I think IO scheduler max bw control should work fine.

BTW, andrea, what is the use case of your patches? Andrew had mentioned that some people are already using it. I am curious to know will a proportional BW controller will solve the issues/requirements of these people or they have specific requirement of traffic shaping and max bw controller only.

[..]

Quoted 18 lines
Quoted 9 lines
Quoted 8 lines
Quoted 2 lines

Can you please give little more details here regarding how QoS requirements are not met with proportional weight?

With proportional weights the whole bandwidth is allocated if no one else is using it. When IO is submitted other tasks with a higher weight can be forced to sleep until the IO generated by the low weight tasks is not completely dispatched. Or any extent of the priority inversion problems.

Quoted 8 lines

Hmm..., I am not very sure here. When admin is allocating the weights, he has the whole picture. He knows how many groups are conteding for the disk and what could be the worst case scenario. So if I have got two groups with A and B with weight 1 and 2 and both are contending, then as an admin one would expect to get 33% of BW for group A in worst case (if group B is continuously backlogged). If B is not contending than A can get 100% of BW. So while configuring the system, will one not plan for worst case (33% for A, and 66 % for B)?

Quoted 7 lines

OK, I'm quite convinced.. :)

To a large degree, if we want to provide a BW reservation strategy we must provide an interface that allows cgroups to ask for time slices such as max/min 5 IO requests every 50ms or something like that. Probably the same functionality can be achieved translating time slices from weights, percentages or absolute BW limits.

Ok, I would like to split it in two parts.

I think providng minimum gurantee in absolute terms like 5 IO request every 50ms will be very hard because IO scheduler has no control over how many competitors are there. An easier thing will be to have minimum gurantees on share basis. For minimum BW (disk time slice) gurantee, admin shall have to create right cgroup hierarchy and assign weights properly and then admin can calculate what % of disk slice a particular group will get as minimum gurantee. (This is more complicated than this as there are time slices which are not accounted to any groups. During queue switch cfq starts the time slice counting only after first request has completed to offset the impact of seeking and i guess also NCQ).

I think it should be possible to give max bandwidth gurantees in absolute terms, like io/s or sectors/sec or MB/sec etc, because only thing IO scheduler has to do is to not allow dispatch from a particular queue if it has crossed its limit and then either let the disk idle or move onto next eligible queue.

Quoted 3 lines

The only issue here will be async writes. max bw gurantee for async writes at IO scheduler level might not mean much to application because of page cache.

I see so much of the memory controller coming up. Since we've been discussing so many of these design points on mail, I wonder if it makes sense to summarize them somewhere (a wiki?). Would anyone like to take a shot at it?

Balbir

#36 lore raw

Re: IO controller discussion (Was: Re: [PATCH 01/10] Documentation)

In reply to a message not in the archive
Quoted 108 lines · Vivek Goyal
Quoted 49 lines · Andrea Righi
Quoted 9 lines · Vivek Goyal
Quoted 6 lines

I think it would be possible to implement both proportional and limiting rules at the same level (e.g., the IO scheduler), but we need also to address the memory consumption problem (I still need to review your patchset in details and I'm going to test it soon :), so I don't know if you already addressed this issue).

Can you please elaborate a bit on this? Are you concerned about that data structures created to solve the problem consume a lot of memory?

Sorry I was not very clear here. With memory consumption I mean wasting the memory with hard/slow reclaimable dirty pages or pending IO requests.

If there's only a global limit on dirty pages, any cgroup can exhaust that limit and cause other cgroups/processes to block when they try to write to disk.

But, ok, the IO controller is not probably the best place to implement such functionality. I should rework on the per cgroup dirty_ratio:

https://lists.linux-foundation.org/pipermail/containers/2008-September/013140.html

Last time we focused too much on the best interfaces to define dirty pages limit, and I never re-posted an updated version of this patchset. Now I think we can simply provide the same dirty_ratio/dirty_bytes interface that we provide globally, but per cgroup.

Quoted 20 lines
Quoted 19 lines

IOW if we simply don't dispatch requests and we don't throttle the tasks in the cgroup that exceeds its limit, how do we avoid the waste of memory due to the succeeding IO requests and the increasingly dirty pages in the page cache (that are also hard to reclaim)? I may be wrong, but I think we talked about this problem in a previous email... sorry I don't find the discussion in my mail archives.

IMHO a nice approach would be to measure IO consumption at the IO scheduler level, and control IO applying proportional weights / absolute limits _both_ at the IO scheduler / elevator level _and_ at the same time block the tasks from dirtying memory that will generate additional IO requests.

Anyway, there's no need to provide this with a single IO controller, we could split the problem in two parts: 1) provide a proportional / absolute IO controller in the IO schedulers and 2) allow to set, for example, a maximum limit of dirty pages for each cgroup.

Quoted 4 lines
Quoted 2 lines

I think setting a maximum limit on dirty pages is an interesting thought. It sounds like as if memory controller can handle it?

Exactly, the same above.

Thinking more about it. Memory controller can probably enforce the higher limit but it would not easily translate into a fixed upper async write rate. Till the process hits the page cache limit or is slowed down by dirty page writeout, it can get a very high async write BW.

So memory controller page cache limit will help but it would not direclty translate into what max bw limit patches are doing.

Even if we do max bw control at IO scheduler level, async writes are problematic again. IO controller will not be able to throttle the process until it sees actuall write request. In big memory systems, writeout might not happen for some time and till then it will see a high throughput.

So doing async write throttling at higher layer and not at IO scheduler layer gives us the opprotunity to produce more accurate results.

For sync requests, I think IO scheduler max bw control should work fine.

BTW, andrea, what is the use case of your patches? Andrew had mentioned that some people are already using it. I am curious to know will a proportional BW controller will solve the issues/requirements of these people or they have specific requirement of traffic shaping and max bw controller only.

[..]

Quoted 18 lines
Quoted 9 lines
Quoted 8 lines
Quoted 2 lines

Can you please give little more details here regarding how QoS requirements are not met with proportional weight?

With proportional weights the whole bandwidth is allocated if no one else is using it. When IO is submitted other tasks with a higher weight can be forced to sleep until the IO generated by the low weight tasks is not completely dispatched. Or any extent of the priority inversion problems.

Quoted 8 lines

Hmm..., I am not very sure here. When admin is allocating the weights, he has the whole picture. He knows how many groups are conteding for the disk and what could be the worst case scenario. So if I have got two groups with A and B with weight 1 and 2 and both are contending, then as an admin one would expect to get 33% of BW for group A in worst case (if group B is continuously backlogged). If B is not contending than A can get 100% of BW. So while configuring the system, will one not plan for worst case (33% for A, and 66 % for B)?

Quoted 7 lines

OK, I'm quite convinced.. :)

To a large degree, if we want to provide a BW reservation strategy we must provide an interface that allows cgroups to ask for time slices such as max/min 5 IO requests every 50ms or something like that. Probably the same functionality can be achieved translating time slices from weights, percentages or absolute BW limits.

Ok, I would like to split it in two parts.

Quoted 10 lines

I think providng minimum gurantee in absolute terms like 5 IO request every 50ms will be very hard because IO scheduler has no control over how many competitors are there. An easier thing will be to have minimum gurantees on share basis. For minimum BW (disk time slice) gurantee, admin shall have to create right cgroup hierarchy and assign weights properly and then admin can calculate what % of disk slice a particular group will get as minimum gurantee. (This is more complicated than this as there are time slices which are not accounted to any groups. During queue switch cfq starts the time slice counting only after first request has completed to offset the impact of seeking and i guess also NCQ).

I agree with Vivek that absolute metrics like 5 IO requests every 50ms might be hard to offer. But 'x ms of disk time every y ms, for a given cgroup' might be a desirable goal. That said, for now we can focus on weight based allocation of disk time, and leave such goals for future.

Quoted 14 lines

I think it should be possible to give max bandwidth gurantees in absolute terms, like io/s or sectors/sec or MB/sec etc, because only thing IO scheduler has to do is to not allow dispatch from a particular queue if it has crossed its limit and then either let the disk idle or move onto next eligible queue.

The only issue here will be async writes. max bw gurantee for async writes at IO scheduler level might not mean much to application because of page cache.

Thanks
Vivek

#37 lore raw

Re: IO controller discussion (Was: Re: [PATCH 01/10] Documentation)

In reply to Vivek Goyal
Quoted 63 lines · Vivek Goyal
Quoted 49 lines · Andrea Righi
Quoted 9 lines · Vivek Goyal
Quoted 6 lines

I think it would be possible to implement both proportional and limiting rules at the same level (e.g., the IO scheduler), but we need also to address the memory consumption problem (I still need to review your patchset in details and I'm going to test it soon :), so I don't know if you already addressed this issue).

Can you please elaborate a bit on this? Are you concerned about that data structures created to solve the problem consume a lot of memory?

Sorry I was not very clear here. With memory consumption I mean wasting the memory with hard/slow reclaimable dirty pages or pending IO requests.

If there's only a global limit on dirty pages, any cgroup can exhaust that limit and cause other cgroups/processes to block when they try to write to disk.

But, ok, the IO controller is not probably the best place to implement such functionality. I should rework on the per cgroup dirty_ratio:

https://lists.linux-foundation.org/pipermail/containers/2008-September/013140.html

Last time we focused too much on the best interfaces to define dirty pages limit, and I never re-posted an updated version of this patchset. Now I think we can simply provide the same dirty_ratio/dirty_bytes interface that we provide globally, but per cgroup.

Quoted 20 lines
Quoted 19 lines

IOW if we simply don't dispatch requests and we don't throttle the tasks in the cgroup that exceeds its limit, how do we avoid the waste of memory due to the succeeding IO requests and the increasingly dirty pages in the page cache (that are also hard to reclaim)? I may be wrong, but I think we talked about this problem in a previous email... sorry I don't find the discussion in my mail archives.

IMHO a nice approach would be to measure IO consumption at the IO scheduler level, and control IO applying proportional weights / absolute limits _both_ at the IO scheduler / elevator level _and_ at the same time block the tasks from dirtying memory that will generate additional IO requests.

Anyway, there's no need to provide this with a single IO controller, we could split the problem in two parts: 1) provide a proportional / absolute IO controller in the IO schedulers and 2) allow to set, for example, a maximum limit of dirty pages for each cgroup.

Quoted 4 lines
Quoted 2 lines

I think setting a maximum limit on dirty pages is an interesting thought. It sounds like as if memory controller can handle it?

Exactly, the same above.

Thinking more about it. Memory controller can probably enforce the higher limit but it would not easily translate into a fixed upper async write rate. Till the process hits the page cache limit or is slowed down by dirty page writeout, it can get a very high async write BW.

So memory controller page cache limit will help but it would not direclty translate into what max bw limit patches are doing.

Quoted 7 lines

Even if we do max bw control at IO scheduler level, async writes are problematic again. IO controller will not be able to throttle the process until it sees actuall write request. In big memory systems, writeout might not happen for some time and till then it will see a high throughput.

So doing async write throttling at higher layer and not at IO scheduler layer gives us the opprotunity to produce more accurate results.

Wouldn't 'doing control on writes at a higher layer' have the same problems as the ones we talk about in dm-ioband? What if the cgroup being throttled for dirtying pages has a high weight assigned to it at the IO scheduler level? What if there are threads of different classes within that cgroup, and we would want to let RT task dirty the pages before BE tasks? I am not sure all these questions make sense, but just wanted to raise issues that might pop up.

If the whole system is designed with cgroups in mind, then throttling at IO scheduler layer should lead to backlog, that could be seen at higher level. For example, if a cgroup is not getting service at IO scheduler level, it should run out of request descriptors, and thus the thread writing back dirty pages should notice it (if its pdflush, blocking it is probably not the best idea). And that should mean the cgroup should hit the dirty threshold, and disallow the task to dirty further pages. There is a possibility though that getting all this right might be an overkill and we can get away with a simpler solution. One possibility seems to be that we provide some feedback from IO scheduling layer to higher layers, that cgroup is hitting its write bandwith limit, and should not be allowed to dirty any more pages.

Quoted 62 lines

For sync requests, I think IO scheduler max bw control should work fine.

BTW, andrea, what is the use case of your patches? Andrew had mentioned that some people are already using it. I am curious to know will a proportional BW controller will solve the issues/requirements of these people or they have specific requirement of traffic shaping and max bw controller only.

[..]

Quoted 18 lines
Quoted 9 lines
Quoted 8 lines
Quoted 2 lines

Can you please give little more details here regarding how QoS requirements are not met with proportional weight?

With proportional weights the whole bandwidth is allocated if no one else is using it. When IO is submitted other tasks with a higher weight can be forced to sleep until the IO generated by the low weight tasks is not completely dispatched. Or any extent of the priority inversion problems.

Quoted 8 lines

Hmm..., I am not very sure here. When admin is allocating the weights, he has the whole picture. He knows how many groups are conteding for the disk and what could be the worst case scenario. So if I have got two groups with A and B with weight 1 and 2 and both are contending, then as an admin one would expect to get 33% of BW for group A in worst case (if group B is continuously backlogged). If B is not contending than A can get 100% of BW. So while configuring the system, will one not plan for worst case (33% for A, and 66 % for B)?

Quoted 7 lines

OK, I'm quite convinced.. :)

To a large degree, if we want to provide a BW reservation strategy we must provide an interface that allows cgroups to ask for time slices such as max/min 5 IO requests every 50ms or something like that. Probably the same functionality can be achieved translating time slices from weights, percentages or absolute BW limits.

Ok, I would like to split it in two parts.

I think providng minimum gurantee in absolute terms like 5 IO request every 50ms will be very hard because IO scheduler has no control over how many competitors are there. An easier thing will be to have minimum gurantees on share basis. For minimum BW (disk time slice) gurantee, admin shall have to create right cgroup hierarchy and assign weights properly and then admin can calculate what % of disk slice a particular group will get as minimum gurantee. (This is more complicated than this as there are time slices which are not accounted to any groups. During queue switch cfq starts the time slice counting only after first request has completed to offset the impact of seeking and i guess also NCQ).

I think it should be possible to give max bandwidth gurantees in absolute terms, like io/s or sectors/sec or MB/sec etc, because only thing IO scheduler has to do is to not allow dispatch from a particular queue if it has crossed its limit and then either let the disk idle or move onto next eligible queue.

The only issue here will be async writes. max bw gurantee for async writes at IO scheduler level might not mean much to application because of page cache.

Thanks
Vivek

A message that is not in the archive (sent off-list or not yet received): e98e18940904171109r17ccb054kb7879f8d02ac26b5-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org

#39 lore raw

Re: IO controller discussion (Was: Re: [PATCH 01/10] Documentation)

Quoted 16 lines · Nauman Rafique
Quoted 8 lines

Thinking more about it. Memory controller can probably enforce the higher limit but it would not easily translate into a fixed upper async write rate. Till the process hits the page cache limit or is slowed down by dirty page writeout, it can get a very high async write BW.

So memory controller page cache limit will help but it would not direclty translate into what max bw limit patches are doing.

Quoted 7 lines

Even if we do max bw control at IO scheduler level, async writes are problematic again. IO controller will not be able to throttle the process until it sees actuall write request. In big memory systems, writeout might not happen for some time and till then it will see a high throughput.

So doing async write throttling at higher layer and not at IO scheduler layer gives us the opprotunity to produce more accurate results.

Quoted 7 lines

Wouldn't 'doing control on writes at a higher layer' have the same problems as the ones we talk about in dm-ioband? What if the cgroup being throttled for dirtying pages has a high weight assigned to it at the IO scheduler level? What if there are threads of different classes within that cgroup, and we would want to let RT task dirty the pages before BE tasks? I am not sure all these questions make sense, but just wanted to raise issues that might pop up.

To a large degree, this seems to be related to provide "fair throttling" at higher level. I mean, throttle equally the tasks belongin to a cgroup that exceeded the limits. With equally I mean proportionally to the IO traffic previously generated _and_ the IO priority.

Otherwise a low priority task doing a lot of IO can consumes all the available cgroup BW and other high priority tasks in the same cgroup may be blocked when they try to write to disk, even if they try to write a small amount of bytes.

Quoted 15 lines

If the whole system is designed with cgroups in mind, then throttling at IO scheduler layer should lead to backlog, that could be seen at higher level. For example, if a cgroup is not getting service at IO scheduler level, it should run out of request descriptors, and thus the thread writing back dirty pages should notice it (if its pdflush, blocking it is probably not the best idea). And that should mean the cgroup should hit the dirty threshold, and disallow the task to dirty further pages. There is a possibility though that getting all this right might be an overkill and we can get away with a simpler solution. One possibility seems to be that we provide some feedback from IO scheduling layer to higher layers, that cgroup is hitting its write bandwith limit, and should not be allowed to dirty any more pages.

IMHO accounting the IO activity in the IO scheduler and blocking the offending application at the higher level is a good solution.

Throttle dirty page ratio could be a nice feature, but probably it's enough to provide a max amount of dirty pages per cgroup and force the tasks to directly writeback those pages when the cgroup exceeded the dirty limit. In this way the dirty page ratio will be automatically throttled by the underlying IO controller.

-Andrea

#40 lore raw

Re: IO controller discussion (Was: Re: [PATCH 01/10] Documentation)

In reply to a message not in the archive
Quoted 80 lines · Nauman Rafique
Quoted 63 lines · Vivek Goyal
Quoted 49 lines · Andrea Righi
Quoted 9 lines · Vivek Goyal
Quoted 6 lines

I think it would be possible to implement both proportional and limiting rules at the same level (e.g., the IO scheduler), but we need also to address the memory consumption problem (I still need to review your patchset in details and I'm going to test it soon :), so I don't know if you already addressed this issue).

Can you please elaborate a bit on this? Are you concerned about that data structures created to solve the problem consume a lot of memory?

Sorry I was not very clear here. With memory consumption I mean wasting the memory with hard/slow reclaimable dirty pages or pending IO requests.

If there's only a global limit on dirty pages, any cgroup can exhaust that limit and cause other cgroups/processes to block when they try to write to disk.

But, ok, the IO controller is not probably the best place to implement such functionality. I should rework on the per cgroup dirty_ratio:

https://lists.linux-foundation.org/pipermail/containers/2008-September/013140.html

Last time we focused too much on the best interfaces to define dirty pages limit, and I never re-posted an updated version of this patchset. Now I think we can simply provide the same dirty_ratio/dirty_bytes interface that we provide globally, but per cgroup.

Quoted 20 lines
Quoted 19 lines

IOW if we simply don't dispatch requests and we don't throttle the tasks in the cgroup that exceeds its limit, how do we avoid the waste of memory due to the succeeding IO requests and the increasingly dirty pages in the page cache (that are also hard to reclaim)? I may be wrong, but I think we talked about this problem in a previous email... sorry I don't find the discussion in my mail archives.

IMHO a nice approach would be to measure IO consumption at the IO scheduler level, and control IO applying proportional weights / absolute limits _both_ at the IO scheduler / elevator level _and_ at the same time block the tasks from dirtying memory that will generate additional IO requests.

Anyway, there's no need to provide this with a single IO controller, we could split the problem in two parts: 1) provide a proportional / absolute IO controller in the IO schedulers and 2) allow to set, for example, a maximum limit of dirty pages for each cgroup.

Quoted 4 lines
Quoted 2 lines

I think setting a maximum limit on dirty pages is an interesting thought. It sounds like as if memory controller can handle it?

Exactly, the same above.

Thinking more about it. Memory controller can probably enforce the higher limit but it would not easily translate into a fixed upper async write rate. Till the process hits the page cache limit or is slowed down by dirty page writeout, it can get a very high async write BW.

So memory controller page cache limit will help but it would not direclty translate into what max bw limit patches are doing.

Quoted 7 lines

Even if we do max bw control at IO scheduler level, async writes are problematic again. IO controller will not be able to throttle the process until it sees actuall write request. In big memory systems, writeout might not happen for some time and till then it will see a high throughput.

So doing async write throttling at higher layer and not at IO scheduler layer gives us the opprotunity to produce more accurate results.

Wouldn't 'doing control on writes at a higher layer' have the same problems as the ones we talk about in dm-ioband? What if the cgroup being throttled for dirtying pages has a high weight assigned to it at the IO scheduler level? What if there are threads of different classes within that cgroup, and we would want to let RT task dirty the pages before BE tasks? I am not sure all these questions make sense, but just wanted to raise issues that might pop up.

Quoted 10 lines

If the whole system is designed with cgroups in mind, then throttling at IO scheduler layer should lead to backlog, that could be seen at higher level. For example, if a cgroup is not getting service at IO scheduler level, it should run out of request descriptors, and thus the thread writing back dirty pages should notice it (if its pdflush, blocking it is probably not the best idea). And that should mean the cgroup should hit the dirty threshold, and disallow the task to dirty further pages. There is a possibility though that getting all this right might be an overkill and we can get away with a simpler solution.

Currently, if pdflush can't keep up and processes are dirtying the page cache at higher rate than we will cross vm_dirty_ratio and process will be made to write back some of dirty pages. That should make sure that processes will be automatically throttled at IO scheduler (Assuming process tries to write its own pages and does not pick randomly some other processes's pages). Currently I think a processes can pick any inode for writeback and not necessarily the inode the process is dirtying.

Thanks
Vivek

#41 lore raw

Re: IO controller discussion (Was: Re: [PATCH 01/10] Documentation)

In reply to Nauman Rafique
Quoted 16 lines · Nauman Rafique
Quoted 8 lines

Thinking more about it. Memory controller can probably enforce the higher limit but it would not easily translate into a fixed upper async write rate. Till the process hits the page cache limit or is slowed down by dirty page writeout, it can get a very high async write BW.

So memory controller page cache limit will help but it would not direclty translate into what max bw limit patches are doing.

Quoted 7 lines

Even if we do max bw control at IO scheduler level, async writes are problematic again. IO controller will not be able to throttle the process until it sees actuall write request. In big memory systems, writeout might not happen for some time and till then it will see a high throughput.

So doing async write throttling at higher layer and not at IO scheduler layer gives us the opprotunity to produce more accurate results.

Quoted 7 lines

Wouldn't 'doing control on writes at a higher layer' have the same problems as the ones we talk about in dm-ioband? What if the cgroup being throttled for dirtying pages has a high weight assigned to it at the IO scheduler level? What if there are threads of different classes within that cgroup, and we would want to let RT task dirty the pages before BE tasks? I am not sure all these questions make sense, but just wanted to raise issues that might pop up.

To a large degree, this seems to be related to provide "fair throttling" at higher level. I mean, throttle equally the tasks belongin to a cgroup that exceeded the limits. With equally I mean proportionally to the IO traffic previously generated _and_ the IO priority.

Otherwise a low priority task doing a lot of IO can consumes all the available cgroup BW and other high priority tasks in the same cgroup may be blocked when they try to write to disk, even if they try to write a small amount of bytes.

Quoted 15 lines

If the whole system is designed with cgroups in mind, then throttling at IO scheduler layer should lead to backlog, that could be seen at higher level. For example, if a cgroup is not getting service at IO scheduler level, it should run out of request descriptors, and thus the thread writing back dirty pages should notice it (if its pdflush, blocking it is probably not the best idea). And that should mean the cgroup should hit the dirty threshold, and disallow the task to dirty further pages. There is a possibility though that getting all this right might be an overkill and we can get away with a simpler solution. One possibility seems to be that we provide some feedback from IO scheduling layer to higher layers, that cgroup is hitting its write bandwith limit, and should not be allowed to dirty any more pages.

IMHO accounting the IO activity in the IO scheduler and blocking the offending application at the higher level is a good solution.

Throttle dirty page ratio could be a nice feature, but probably it's enough to provide a max amount of dirty pages per cgroup and force the tasks to directly writeback those pages when the cgroup exceeded the dirty limit. In this way the dirty page ratio will be automatically throttled by the underlying IO controller.

-Andrea

#42 lore raw

Re: IO controller discussion (Was: Re: [PATCH 01/10] Documentation)

In reply to Nauman Rafique
Quoted 72 lines · Nauman Rafique
Quoted 63 lines · Vivek Goyal
Quoted 49 lines · Andrea Righi
Quoted 9 lines · Vivek Goyal
Quoted 6 lines

I think it would be possible to implement both proportional and limiting rules at the same level (e.g., the IO scheduler), but we need also to address the memory consumption problem (I still need to review your patchset in details and I'm going to test it soon :), so I don't know if you already addressed this issue).

Can you please elaborate a bit on this? Are you concerned about that data structures created to solve the problem consume a lot of memory?

Sorry I was not very clear here. With memory consumption I mean wasting the memory with hard/slow reclaimable dirty pages or pending IO requests.

If there's only a global limit on dirty pages, any cgroup can exhaust that limit and cause other cgroups/processes to block when they try to write to disk.

But, ok, the IO controller is not probably the best place to implement such functionality. I should rework on the per cgroup dirty_ratio:

https://lists.linux-foundation.org/pipermail/containers/2008-September/013140.html

Last time we focused too much on the best interfaces to define dirty pages limit, and I never re-posted an updated version of this patchset. Now I think we can simply provide the same dirty_ratio/dirty_bytes interface that we provide globally, but per cgroup.

Quoted 20 lines
Quoted 19 lines

IOW if we simply don't dispatch requests and we don't throttle the tasks in the cgroup that exceeds its limit, how do we avoid the waste of memory due to the succeeding IO requests and the increasingly dirty pages in the page cache (that are also hard to reclaim)? I may be wrong, but I think we talked about this problem in a previous email... sorry I don't find the discussion in my mail archives.

IMHO a nice approach would be to measure IO consumption at the IO scheduler level, and control IO applying proportional weights / absolute limits _both_ at the IO scheduler / elevator level _and_ at the same time block the tasks from dirtying memory that will generate additional IO requests.

Anyway, there's no need to provide this with a single IO controller, we could split the problem in two parts: 1) provide a proportional / absolute IO controller in the IO schedulers and 2) allow to set, for example, a maximum limit of dirty pages for each cgroup.

Quoted 4 lines
Quoted 2 lines

I think setting a maximum limit on dirty pages is an interesting thought. It sounds like as if memory controller can handle it?

Exactly, the same above.

Thinking more about it. Memory controller can probably enforce the higher limit but it would not easily translate into a fixed upper async write rate. Till the process hits the page cache limit or is slowed down by dirty page writeout, it can get a very high async write BW.

So memory controller page cache limit will help but it would not direclty translate into what max bw limit patches are doing.

Quoted 7 lines

Even if we do max bw control at IO scheduler level, async writes are problematic again. IO controller will not be able to throttle the process until it sees actuall write request. In big memory systems, writeout might not happen for some time and till then it will see a high throughput.

So doing async write throttling at higher layer and not at IO scheduler layer gives us the opprotunity to produce more accurate results.

Quoted 8 lines

Wouldn't 'doing control on writes at a higher layer' have the same problems as the ones we talk about in dm-ioband? What if the cgroup being throttled for dirtying pages has a high weight assigned to it at the IO scheduler level? What if there are threads of different classes within that cgroup, and we would want to let RT task dirty the pages before BE tasks? I am not sure all these questions make sense, but just wanted to raise issues that might pop up.

Yes, i would think that you will run into same issues if one is not maintaining separate queues. The only difference is that we are throttling only async writes and i am not sure how well the notion of fairness is defined for async writes in current system as most of the time nobody has cared too much for latency of async writes (fsync is the exception).

But in general, yes, putting things into a single queue again puts into the same situation where we loose the notion of class and prio with-in cgroup. So any design which tries to do that probably is not a very good idea.

Quoted 5 lines

If the whole system is designed with cgroups in mind, then throttling at IO scheduler layer should lead to backlog, that could be seen at higher level. For example, if a cgroup is not getting service at IO scheduler level, it should run out of request descriptors, and thus the thread writing back dirty pages should notice it (if its pdflush,

Quoted 8 lines

blocking it is probably not the best idea). And that should mean the cgroup should hit the dirty threshold, and disallow the task to dirty further pages. There is a possibility though that getting all this right might be an overkill and we can get away with a simpler solution. One possibility seems to be that we provide some feedback from IO scheduling layer to higher layers, that cgroup is hitting its write bandwith limit, and should not be allowed to dirty any more pages.

I think bdi_write_congested(), already provides the feedback to upper layer about how congested the queue is and it is likely that any thread submitting new write request to this bdi will be made to sleep before it gets the reqeust descriptor.

I think in new scheme of thing, this notion of per device congestion shall have to changed into per device per cgroup congestion.

Thanks
Vivek

Quoted 63 lines
Quoted 63 lines

For sync requests, I think IO scheduler max bw control should work fine.

BTW, andrea, what is the use case of your patches? Andrew had mentioned that some people are already using it. I am curious to know will a proportional BW controller will solve the issues/requirements of these people or they have specific requirement of traffic shaping and max bw controller only.

[..]

Quoted 18 lines
Quoted 9 lines
Quoted 8 lines
Quoted 2 lines

Can you please give little more details here regarding how QoS requirements are not met with proportional weight?

With proportional weights the whole bandwidth is allocated if no one else is using it. When IO is submitted other tasks with a higher weight can be forced to sleep until the IO generated by the low weight tasks is not completely dispatched. Or any extent of the priority inversion problems.

Quoted 8 lines

Hmm..., I am not very sure here. When admin is allocating the weights, he has the whole picture. He knows how many groups are conteding for the disk and what could be the worst case scenario. So if I have got two groups with A and B with weight 1 and 2 and both are contending, then as an admin one would expect to get 33% of BW for group A in worst case (if group B is continuously backlogged). If B is not contending than A can get 100% of BW. So while configuring the system, will one not plan for worst case (33% for A, and 66 % for B)?

Quoted 7 lines

OK, I'm quite convinced.. :)

To a large degree, if we want to provide a BW reservation strategy we must provide an interface that allows cgroups to ask for time slices such as max/min 5 IO requests every 50ms or something like that. Probably the same functionality can be achieved translating time slices from weights, percentages or absolute BW limits.

Ok, I would like to split it in two parts.

I think providng minimum gurantee in absolute terms like 5 IO request every 50ms will be very hard because IO scheduler has no control over how many competitors are there. An easier thing will be to have minimum gurantees on share basis. For minimum BW (disk time slice) gurantee, admin shall have to create right cgroup hierarchy and assign weights properly and then admin can calculate what % of disk slice a particular group will get as minimum gurantee. (This is more complicated than this as there are time slices which are not accounted to any groups. During queue switch cfq starts the time slice counting only after first request has completed to offset the impact of seeking and i guess also NCQ).

I think it should be possible to give max bandwidth gurantees in absolute terms, like io/s or sectors/sec or MB/sec etc, because only thing IO scheduler has to do is to not allow dispatch from a particular queue if it has crossed its limit and then either let the disk idle or move onto next eligible queue.

The only issue here will be async writes. max bw gurantee for async writes at IO scheduler level might not mean much to application because of page cache.

Thanks
Vivek

#43 lore raw

Re: IO controller discussion (Was: Re: [PATCH 01/10] Documentation)

In reply to Nauman Rafique
Quoted 80 lines · Nauman Rafique
Quoted 63 lines · Vivek Goyal
Quoted 49 lines · Andrea Righi
Quoted 9 lines · Vivek Goyal
Quoted 6 lines

I think it would be possible to implement both proportional and limiting rules at the same level (e.g., the IO scheduler), but we need also to address the memory consumption problem (I still need to review your patchset in details and I'm going to test it soon :), so I don't know if you already addressed this issue).

Can you please elaborate a bit on this? Are you concerned about that data structures created to solve the problem consume a lot of memory?

Sorry I was not very clear here. With memory consumption I mean wasting the memory with hard/slow reclaimable dirty pages or pending IO requests.

If there's only a global limit on dirty pages, any cgroup can exhaust that limit and cause other cgroups/processes to block when they try to write to disk.

But, ok, the IO controller is not probably the best place to implement such functionality. I should rework on the per cgroup dirty_ratio:

https://lists.linux-foundation.org/pipermail/containers/2008-September/013140.html

Last time we focused too much on the best interfaces to define dirty pages limit, and I never re-posted an updated version of this patchset. Now I think we can simply provide the same dirty_ratio/dirty_bytes interface that we provide globally, but per cgroup.

Quoted 20 lines
Quoted 19 lines

IOW if we simply don't dispatch requests and we don't throttle the tasks in the cgroup that exceeds its limit, how do we avoid the waste of memory due to the succeeding IO requests and the increasingly dirty pages in the page cache (that are also hard to reclaim)? I may be wrong, but I think we talked about this problem in a previous email... sorry I don't find the discussion in my mail archives.

IMHO a nice approach would be to measure IO consumption at the IO scheduler level, and control IO applying proportional weights / absolute limits _both_ at the IO scheduler / elevator level _and_ at the same time block the tasks from dirtying memory that will generate additional IO requests.

Anyway, there's no need to provide this with a single IO controller, we could split the problem in two parts: 1) provide a proportional / absolute IO controller in the IO schedulers and 2) allow to set, for example, a maximum limit of dirty pages for each cgroup.

Quoted 4 lines
Quoted 2 lines

I think setting a maximum limit on dirty pages is an interesting thought. It sounds like as if memory controller can handle it?

Exactly, the same above.

Thinking more about it. Memory controller can probably enforce the higher limit but it would not easily translate into a fixed upper async write rate. Till the process hits the page cache limit or is slowed down by dirty page writeout, it can get a very high async write BW.

So memory controller page cache limit will help but it would not direclty translate into what max bw limit patches are doing.

Quoted 7 lines

Even if we do max bw control at IO scheduler level, async writes are problematic again. IO controller will not be able to throttle the process until it sees actuall write request. In big memory systems, writeout might not happen for some time and till then it will see a high throughput.

So doing async write throttling at higher layer and not at IO scheduler layer gives us the opprotunity to produce more accurate results.

Wouldn't 'doing control on writes at a higher layer' have the same problems as the ones we talk about in dm-ioband? What if the cgroup being throttled for dirtying pages has a high weight assigned to it at the IO scheduler level? What if there are threads of different classes within that cgroup, and we would want to let RT task dirty the pages before BE tasks? I am not sure all these questions make sense, but just wanted to raise issues that might pop up.

Quoted 10 lines

If the whole system is designed with cgroups in mind, then throttling at IO scheduler layer should lead to backlog, that could be seen at higher level. For example, if a cgroup is not getting service at IO scheduler level, it should run out of request descriptors, and thus the thread writing back dirty pages should notice it (if its pdflush, blocking it is probably not the best idea). And that should mean the cgroup should hit the dirty threshold, and disallow the task to dirty further pages. There is a possibility though that getting all this right might be an overkill and we can get away with a simpler solution.

Currently, if pdflush can't keep up and processes are dirtying the page cache at higher rate than we will cross vm_dirty_ratio and process will be made to write back some of dirty pages. That should make sure that processes will be automatically throttled at IO scheduler (Assuming process tries to write its own pages and does not pick randomly some other processes's pages). Currently I think a processes can pick any inode for writeback and not necessarily the inode the process is dirtying.

Thanks
Vivek

#44 lore raw

Re: IO controller discussion (Was: Re: [PATCH 01/10] Documentation)

In reply to Vivek Goyal
Quoted 5 lines · Vivek Goyal
Quoted 4 lines
Quoted 2 lines

I think setting a maximum limit on dirty pages is an interesting thought. It sounds like as if memory controller can handle it?

Exactly, the same above.

Quoted 7 lines

Thinking more about it. Memory controller can probably enforce the higher limit but it would not easily translate into a fixed upper async write rate. Till the process hits the page cache limit or is slowed down by dirty page writeout, it can get a very high async write BW.

So memory controller page cache limit will help but it would not direclty translate into what max bw limit patches are doing.

The memory controller can be used to set an upper limit of the dirty pages. When this limit is exceeded the tasks in the cgroup can be forced to write the exceeding dirty pages to disk. At this point the IO controller can: 1) throttle the task that is going to submit the IO requests, if the guy that dirtied the pages was actually the task itself, or 2) delay the submission of those requests to the elevator (or at the IO scheduler level) if it's writeback IO (e.g., made by pdflush).

Both functionalities should allow to have a BW control and avoid that any single cgroup can entirely exhaust the global limit of dirty pages.

Quoted 8 lines

Even if we do max bw control at IO scheduler level, async writes are problematic again. IO controller will not be able to throttle the process until it sees actuall write request. In big memory systems, writeout might not happen for some time and till then it will see a high throughput.

So doing async write throttling at higher layer and not at IO scheduler layer gives us the opprotunity to produce more accurate results.

Totally agree.

Quoted 2 lines

For sync requests, I think IO scheduler max bw control should work fine.

ditto

-Andrea

#45 lore raw

Re: IO controller discussion (Was: Re: [PATCH 01/10] Documentation)

Quoted 14 lines · Andrea Righi
Quoted 5 lines · Vivek Goyal
Quoted 4 lines
Quoted 2 lines

I think setting a maximum limit on dirty pages is an interesting thought. It sounds like as if memory controller can handle it?

Exactly, the same above.

Quoted 7 lines

Thinking more about it. Memory controller can probably enforce the higher limit but it would not easily translate into a fixed upper async write rate. Till the process hits the page cache limit or is slowed down by dirty page writeout, it can get a very high async write BW.

So memory controller page cache limit will help but it would not direclty translate into what max bw limit patches are doing.

Quoted 8 lines

The memory controller can be used to set an upper limit of the dirty pages. When this limit is exceeded the tasks in the cgroup can be forced to write the exceeding dirty pages to disk. At this point the IO controller can: 1) throttle the task that is going to submit the IO requests, if the guy that dirtied the pages was actually the task itself, or 2) delay the submission of those requests to the elevator (or at the IO scheduler level) if it's writeback IO (e.g., made by pdflush).

True, per cgroup dirty pages limit will help in making sure one cgroup does not run away mojority share of the page cache. And once a cgroup hits dirty limit it is forced to do write back.

But my point is that it hels in bandwidth control but it does not directly translate into what max bw patches are doing. I thought your goal with max bw patches was to provide the consistent upper limit on BW seem by the application. So till an application hits the per cgroup dirty limit, it might see an spike in async write BW (much more than what has been specified by per cgroup max bw limit) and that will defeat the purpose of max bw controller up to some extent?

Quoted 9 lines

Both functionalities should allow to have a BW control and avoid that any single cgroup can entirely exhaust the global limit of dirty pages.

Quoted 6 lines

Even if we do max bw control at IO scheduler level, async writes are problematic again. IO controller will not be able to throttle the process until it sees actuall write request. In big memory systems, writeout might not happen for some time and till then it will see a high throughput.

Quoted 4 lines
Quoted 2 lines

So doing async write throttling at higher layer and not at IO scheduler layer gives us the opprotunity to produce more accurate results.

Totally agree.

I will correct myself here. After going through the documentation of max bw controller patches, it looks like that you are also controlling async writes only after they are actually being written to the disk and not at the time of async write admission in page cache.

If that's the case then doing this control at IO scheduler level should produce the similar results what you are seeing now with higher level control. In fact throttling at IO scheduler has advantage that one does not have to worry about maintaining multiple queues for separate class and prio requests as IO scheduler already does it.

Thanks
Vivek

Quoted 7 lines
Quoted 3 lines

For sync requests, I think IO scheduler max bw control should work fine.

ditto

-Andrea

#46 lore raw

Re: IO controller discussion (Was: Re: [PATCH 01/10] Documentation)

In reply to Vivek Goyal
Quoted 125 lines · Vivek Goyal
Quoted 49 lines · Andrea Righi
Quoted 9 lines · Vivek Goyal
Quoted 6 lines

I think it would be possible to implement both proportional and limiting rules at the same level (e.g., the IO scheduler), but we need also to address the memory consumption problem (I still need to review your patchset in details and I'm going to test it soon :), so I don't know if you already addressed this issue).

Can you please elaborate a bit on this? Are you concerned about that data structures created to solve the problem consume a lot of memory?

Sorry I was not very clear here. With memory consumption I mean wasting the memory with hard/slow reclaimable dirty pages or pending IO requests.

If there's only a global limit on dirty pages, any cgroup can exhaust that limit and cause other cgroups/processes to block when they try to write to disk.

But, ok, the IO controller is not probably the best place to implement such functionality. I should rework on the per cgroup dirty_ratio:

https://lists.linux-foundation.org/pipermail/containers/2008-September/013140.html

Last time we focused too much on the best interfaces to define dirty pages limit, and I never re-posted an updated version of this patchset. Now I think we can simply provide the same dirty_ratio/dirty_bytes interface that we provide globally, but per cgroup.

Quoted 20 lines
Quoted 19 lines

IOW if we simply don't dispatch requests and we don't throttle the tasks in the cgroup that exceeds its limit, how do we avoid the waste of memory due to the succeeding IO requests and the increasingly dirty pages in the page cache (that are also hard to reclaim)? I may be wrong, but I think we talked about this problem in a previous email... sorry I don't find the discussion in my mail archives.

IMHO a nice approach would be to measure IO consumption at the IO scheduler level, and control IO applying proportional weights / absolute limits _both_ at the IO scheduler / elevator level _and_ at the same time block the tasks from dirtying memory that will generate additional IO requests.

Anyway, there's no need to provide this with a single IO controller, we could split the problem in two parts: 1) provide a proportional / absolute IO controller in the IO schedulers and 2) allow to set, for example, a maximum limit of dirty pages for each cgroup.

Quoted 4 lines
Quoted 2 lines

I think setting a maximum limit on dirty pages is an interesting thought. It sounds like as if memory controller can handle it?

Exactly, the same above.

Thinking more about it. Memory controller can probably enforce the higher limit but it would not easily translate into a fixed upper async write rate. Till the process hits the page cache limit or is slowed down by dirty page writeout, it can get a very high async write BW.

So memory controller page cache limit will help but it would not direclty translate into what max bw limit patches are doing.

Even if we do max bw control at IO scheduler level, async writes are problematic again. IO controller will not be able to throttle the process until it sees actuall write request. In big memory systems, writeout might not happen for some time and till then it will see a high throughput.

So doing async write throttling at higher layer and not at IO scheduler layer gives us the opprotunity to produce more accurate results.

For sync requests, I think IO scheduler max bw control should work fine.

BTW, andrea, what is the use case of your patches? Andrew had mentioned that some people are already using it. I am curious to know will a proportional BW controller will solve the issues/requirements of these people or they have specific requirement of traffic shaping and max bw controller only.

[..]

Quoted 18 lines
Quoted 9 lines
Quoted 8 lines
Quoted 2 lines

Can you please give little more details here regarding how QoS requirements are not met with proportional weight?

With proportional weights the whole bandwidth is allocated if no one else is using it. When IO is submitted other tasks with a higher weight can be forced to sleep until the IO generated by the low weight tasks is not completely dispatched. Or any extent of the priority inversion problems.

Quoted 8 lines

Hmm..., I am not very sure here. When admin is allocating the weights, he has the whole picture. He knows how many groups are conteding for the disk and what could be the worst case scenario. So if I have got two groups with A and B with weight 1 and 2 and both are contending, then as an admin one would expect to get 33% of BW for group A in worst case (if group B is continuously backlogged). If B is not contending than A can get 100% of BW. So while configuring the system, will one not plan for worst case (33% for A, and 66 % for B)?

Quoted 7 lines

OK, I'm quite convinced.. :)

To a large degree, if we want to provide a BW reservation strategy we must provide an interface that allows cgroups to ask for time slices such as max/min 5 IO requests every 50ms or something like that. Probably the same functionality can be achieved translating time slices from weights, percentages or absolute BW limits.

Ok, I would like to split it in two parts.

I think providng minimum gurantee in absolute terms like 5 IO request every 50ms will be very hard because IO scheduler has no control over how many competitors are there. An easier thing will be to have minimum gurantees on share basis. For minimum BW (disk time slice) gurantee, admin shall have to create right cgroup hierarchy and assign weights properly and then admin can calculate what % of disk slice a particular group will get as minimum gurantee. (This is more complicated than this as there are time slices which are not accounted to any groups. During queue switch cfq starts the time slice counting only after first request has completed to offset the impact of seeking and i guess also NCQ).

I think it should be possible to give max bandwidth gurantees in absolute terms, like io/s or sectors/sec or MB/sec etc, because only thing IO scheduler has to do is to not allow dispatch from a particular queue if it has crossed its limit and then either let the disk idle or move onto next eligible queue.

Quoted 3 lines

The only issue here will be async writes. max bw gurantee for async writes at IO scheduler level might not mean much to application because of page cache.

I see so much of the memory controller coming up. Since we've been discussing so many of these design points on mail, I wonder if it makes sense to summarize them somewhere (a wiki?). Would anyone like to take a shot at it?

Balbir

A message that is not in the archive (sent off-list or not yet received): 661de9470904180619k34e7998ch755a2ad3bed9ce5e-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org

#48 lore raw

Re: IO controller discussion (Was: Re: [PATCH 01/10] Documentation)

Quoted 126 lines · Balbir Singh
Quoted 125 lines · Vivek Goyal
Quoted 49 lines · Andrea Righi
Quoted 9 lines · Vivek Goyal
Quoted 6 lines

I think it would be possible to implement both proportional and limiting rules at the same level (e.g., the IO scheduler), but we need also to address the memory consumption problem (I still need to review your patchset in details and I'm going to test it soon :), so I don't know if you already addressed this issue).

Can you please elaborate a bit on this? Are you concerned about that data structures created to solve the problem consume a lot of memory?

Sorry I was not very clear here. With memory consumption I mean wasting the memory with hard/slow reclaimable dirty pages or pending IO requests.

If there's only a global limit on dirty pages, any cgroup can exhaust that limit and cause other cgroups/processes to block when they try to write to disk.

But, ok, the IO controller is not probably the best place to implement such functionality. I should rework on the per cgroup dirty_ratio:

https://lists.linux-foundation.org/pipermail/containers/2008-September/013140.html

Last time we focused too much on the best interfaces to define dirty pages limit, and I never re-posted an updated version of this patchset. Now I think we can simply provide the same dirty_ratio/dirty_bytes interface that we provide globally, but per cgroup.

Quoted 20 lines
Quoted 19 lines

IOW if we simply don't dispatch requests and we don't throttle the tasks in the cgroup that exceeds its limit, how do we avoid the waste of memory due to the succeeding IO requests and the increasingly dirty pages in the page cache (that are also hard to reclaim)? I may be wrong, but I think we talked about this problem in a previous email... sorry I don't find the discussion in my mail archives.

IMHO a nice approach would be to measure IO consumption at the IO scheduler level, and control IO applying proportional weights / absolute limits _both_ at the IO scheduler / elevator level _and_ at the same time block the tasks from dirtying memory that will generate additional IO requests.

Anyway, there's no need to provide this with a single IO controller, we could split the problem in two parts: 1) provide a proportional / absolute IO controller in the IO schedulers and 2) allow to set, for example, a maximum limit of dirty pages for each cgroup.

Quoted 4 lines
Quoted 2 lines

I think setting a maximum limit on dirty pages is an interesting thought. It sounds like as if memory controller can handle it?

Exactly, the same above.

Thinking more about it. Memory controller can probably enforce the higher limit but it would not easily translate into a fixed upper async write rate. Till the process hits the page cache limit or is slowed down by dirty page writeout, it can get a very high async write BW.

So memory controller page cache limit will help but it would not direclty translate into what max bw limit patches are doing.

Even if we do max bw control at IO scheduler level, async writes are problematic again. IO controller will not be able to throttle the process until it sees actuall write request. In big memory systems, writeout might not happen for some time and till then it will see a high throughput.

So doing async write throttling at higher layer and not at IO scheduler layer gives us the opprotunity to produce more accurate results.

For sync requests, I think IO scheduler max bw control should work fine.

BTW, andrea, what is the use case of your patches? Andrew had mentioned that some people are already using it. I am curious to know will a proportional BW controller will solve the issues/requirements of these people or they have specific requirement of traffic shaping and max bw controller only.

[..]

Quoted 18 lines
Quoted 9 lines
Quoted 8 lines
Quoted 2 lines

Can you please give little more details here regarding how QoS requirements are not met with proportional weight?

With proportional weights the whole bandwidth is allocated if no one else is using it. When IO is submitted other tasks with a higher weight can be forced to sleep until the IO generated by the low weight tasks is not completely dispatched. Or any extent of the priority inversion problems.

Quoted 8 lines

Hmm..., I am not very sure here. When admin is allocating the weights, he has the whole picture. He knows how many groups are conteding for the disk and what could be the worst case scenario. So if I have got two groups with A and B with weight 1 and 2 and both are contending, then as an admin one would expect to get 33% of BW for group A in worst case (if group B is continuously backlogged). If B is not contending than A can get 100% of BW. So while configuring the system, will one not plan for worst case (33% for A, and 66 % for B)?

Quoted 7 lines

OK, I'm quite convinced.. :)

To a large degree, if we want to provide a BW reservation strategy we must provide an interface that allows cgroups to ask for time slices such as max/min 5 IO requests every 50ms or something like that. Probably the same functionality can be achieved translating time slices from weights, percentages or absolute BW limits.

Ok, I would like to split it in two parts.

I think providng minimum gurantee in absolute terms like 5 IO request every 50ms will be very hard because IO scheduler has no control over how many competitors are there. An easier thing will be to have minimum gurantees on share basis. For minimum BW (disk time slice) gurantee, admin shall have to create right cgroup hierarchy and assign weights properly and then admin can calculate what % of disk slice a particular group will get as minimum gurantee. (This is more complicated than this as there are time slices which are not accounted to any groups. During queue switch cfq starts the time slice counting only after first request has completed to offset the impact of seeking and i guess also NCQ).

I think it should be possible to give max bandwidth gurantees in absolute terms, like io/s or sectors/sec or MB/sec etc, because only thing IO scheduler has to do is to not allow dispatch from a particular queue if it has crossed its limit and then either let the disk idle or move onto next eligible queue.

Quoted 8 lines
Quoted 3 lines

The only issue here will be async writes. max bw gurantee for async writes at IO scheduler level might not mean much to application because of page cache.

I see so much of the memory controller coming up. Since we've been discussing so many of these design points on mail, I wonder if it makes sense to summarize them somewhere (a wiki?). Would anyone like to take a shot at it?

Balbir, this is definitely a good idea. Just that once we have had some more discussion and some sort of understanding of issues, it might make more sense.

Got a question for you. Does memory controller already have the per cgroup dirty pages limit? If no, has this been discussed in the past? if yes, what was the conclsion?

Thanks
Vivek

#49 lore raw

Re: IO controller discussion (Was: Re: [PATCH 01/10] Documentation)

In reply to Balbir Singh
Quoted 126 lines · Balbir Singh
Quoted 125 lines · Vivek Goyal
Quoted 49 lines · Andrea Righi
Quoted 9 lines · Vivek Goyal
Quoted 6 lines

I think it would be possible to implement both proportional and limiting rules at the same level (e.g., the IO scheduler), but we need also to address the memory consumption problem (I still need to review your patchset in details and I'm going to test it soon :), so I don't know if you already addressed this issue).

Can you please elaborate a bit on this? Are you concerned about that data structures created to solve the problem consume a lot of memory?

Sorry I was not very clear here. With memory consumption I mean wasting the memory with hard/slow reclaimable dirty pages or pending IO requests.

If there's only a global limit on dirty pages, any cgroup can exhaust that limit and cause other cgroups/processes to block when they try to write to disk.

But, ok, the IO controller is not probably the best place to implement such functionality. I should rework on the per cgroup dirty_ratio:

https://lists.linux-foundation.org/pipermail/containers/2008-September/013140.html

Last time we focused too much on the best interfaces to define dirty pages limit, and I never re-posted an updated version of this patchset. Now I think we can simply provide the same dirty_ratio/dirty_bytes interface that we provide globally, but per cgroup.

Quoted 20 lines
Quoted 19 lines

IOW if we simply don't dispatch requests and we don't throttle the tasks in the cgroup that exceeds its limit, how do we avoid the waste of memory due to the succeeding IO requests and the increasingly dirty pages in the page cache (that are also hard to reclaim)? I may be wrong, but I think we talked about this problem in a previous email... sorry I don't find the discussion in my mail archives.

IMHO a nice approach would be to measure IO consumption at the IO scheduler level, and control IO applying proportional weights / absolute limits _both_ at the IO scheduler / elevator level _and_ at the same time block the tasks from dirtying memory that will generate additional IO requests.

Anyway, there's no need to provide this with a single IO controller, we could split the problem in two parts: 1) provide a proportional / absolute IO controller in the IO schedulers and 2) allow to set, for example, a maximum limit of dirty pages for each cgroup.

Quoted 4 lines
Quoted 2 lines

I think setting a maximum limit on dirty pages is an interesting thought. It sounds like as if memory controller can handle it?

Exactly, the same above.

Thinking more about it. Memory controller can probably enforce the higher limit but it would not easily translate into a fixed upper async write rate. Till the process hits the page cache limit or is slowed down by dirty page writeout, it can get a very high async write BW.

So memory controller page cache limit will help but it would not direclty translate into what max bw limit patches are doing.

Even if we do max bw control at IO scheduler level, async writes are problematic again. IO controller will not be able to throttle the process until it sees actuall write request. In big memory systems, writeout might not happen for some time and till then it will see a high throughput.

So doing async write throttling at higher layer and not at IO scheduler layer gives us the opprotunity to produce more accurate results.

For sync requests, I think IO scheduler max bw control should work fine.

BTW, andrea, what is the use case of your patches? Andrew had mentioned that some people are already using it. I am curious to know will a proportional BW controller will solve the issues/requirements of these people or they have specific requirement of traffic shaping and max bw controller only.

[..]

Quoted 18 lines
Quoted 9 lines
Quoted 8 lines
Quoted 2 lines

Can you please give little more details here regarding how QoS requirements are not met with proportional weight?

With proportional weights the whole bandwidth is allocated if no one else is using it. When IO is submitted other tasks with a higher weight can be forced to sleep until the IO generated by the low weight tasks is not completely dispatched. Or any extent of the priority inversion problems.

Quoted 8 lines

Hmm..., I am not very sure here. When admin is allocating the weights, he has the whole picture. He knows how many groups are conteding for the disk and what could be the worst case scenario. So if I have got two groups with A and B with weight 1 and 2 and both are contending, then as an admin one would expect to get 33% of BW for group A in worst case (if group B is continuously backlogged). If B is not contending than A can get 100% of BW. So while configuring the system, will one not plan for worst case (33% for A, and 66 % for B)?

Quoted 7 lines

OK, I'm quite convinced.. :)

To a large degree, if we want to provide a BW reservation strategy we must provide an interface that allows cgroups to ask for time slices such as max/min 5 IO requests every 50ms or something like that. Probably the same functionality can be achieved translating time slices from weights, percentages or absolute BW limits.

Ok, I would like to split it in two parts.

I think providng minimum gurantee in absolute terms like 5 IO request every 50ms will be very hard because IO scheduler has no control over how many competitors are there. An easier thing will be to have minimum gurantees on share basis. For minimum BW (disk time slice) gurantee, admin shall have to create right cgroup hierarchy and assign weights properly and then admin can calculate what % of disk slice a particular group will get as minimum gurantee. (This is more complicated than this as there are time slices which are not accounted to any groups. During queue switch cfq starts the time slice counting only after first request has completed to offset the impact of seeking and i guess also NCQ).

I think it should be possible to give max bandwidth gurantees in absolute terms, like io/s or sectors/sec or MB/sec etc, because only thing IO scheduler has to do is to not allow dispatch from a particular queue if it has crossed its limit and then either let the disk idle or move onto next eligible queue.

Quoted 8 lines
Quoted 3 lines

The only issue here will be async writes. max bw gurantee for async writes at IO scheduler level might not mean much to application because of page cache.

I see so much of the memory controller coming up. Since we've been discussing so many of these design points on mail, I wonder if it makes sense to summarize them somewhere (a wiki?). Would anyone like to take a shot at it?

Balbir, this is definitely a good idea. Just that once we have had some more discussion and some sort of understanding of issues, it might make more sense.

Got a question for you. Does memory controller already have the per cgroup dirty pages limit? If no, has this been discussed in the past? if yes, what was the conclsion?

Thanks
Vivek

A message that is not in the archive (sent off-list or not yet received): [email protected]

#51 lore raw

Re: IO controller discussion (Was: Re: [PATCH 01/10] Documentation)

Quoted 131 lines · Vivek Goyal
Quoted 130 lines · Balbir Singh
Quoted 128 lines · Vivek Goyal
Quoted 49 lines · Andrea Righi
Quoted 9 lines · Vivek Goyal
Quoted 6 lines

I think it would be possible to implement both proportional and limiting rules at the same level (e.g., the IO scheduler), but we need also to address the memory consumption problem (I still need to review your patchset in details and I'm going to test it soon :), so I don't know if you already addressed this issue).

Can you please elaborate a bit on this? Are you concerned about that data structures created to solve the problem consume a lot of memory?

Sorry I was not very clear here. With memory consumption I mean wasting the memory with hard/slow reclaimable dirty pages or pending IO requests.

If there's only a global limit on dirty pages, any cgroup can exhaust that limit and cause other cgroups/processes to block when they try to write to disk.

But, ok, the IO controller is not probably the best place to implement such functionality. I should rework on the per cgroup dirty_ratio:

https://lists.linux-foundation.org/pipermail/containers/2008-September/013140.html

Last time we focused too much on the best interfaces to define dirty pages limit, and I never re-posted an updated version of this patchset. Now I think we can simply provide the same dirty_ratio/dirty_bytes interface that we provide globally, but per cgroup.

Quoted 20 lines
Quoted 19 lines

IOW if we simply don't dispatch requests and we don't throttle the tasks in the cgroup that exceeds its limit, how do we avoid the waste of memory due to the succeeding IO requests and the increasingly dirty pages in the page cache (that are also hard to reclaim)? I may be wrong, but I think we talked about this problem in a previous email... sorry I don't find the discussion in my mail archives.

IMHO a nice approach would be to measure IO consumption at the IO scheduler level, and control IO applying proportional weights / absolute limits _both_ at the IO scheduler / elevator level _and_ at the same time block the tasks from dirtying memory that will generate additional IO requests.

Anyway, there's no need to provide this with a single IO controller, we could split the problem in two parts: 1) provide a proportional / absolute IO controller in the IO schedulers and 2) allow to set, for example, a maximum limit of dirty pages for each cgroup.

Quoted 4 lines
Quoted 2 lines

I think setting a maximum limit on dirty pages is an interesting thought. It sounds like as if memory controller can handle it?

Exactly, the same above.

Thinking more about it. Memory controller can probably enforce the higher limit but it would not easily translate into a fixed upper async write rate. Till the process hits the page cache limit or is slowed down by dirty page writeout, it can get a very high async write BW.

So memory controller page cache limit will help but it would not direclty translate into what max bw limit patches are doing.

Even if we do max bw control at IO scheduler level, async writes are problematic again. IO controller will not be able to throttle the process until it sees actuall write request. In big memory systems, writeout might not happen for some time and till then it will see a high throughput.

So doing async write throttling at higher layer and not at IO scheduler layer gives us the opprotunity to produce more accurate results.

For sync requests, I think IO scheduler max bw control should work fine.

BTW, andrea, what is the use case of your patches? Andrew had mentioned that some people are already using it. I am curious to know will a proportional BW controller will solve the issues/requirements of these people or they have specific requirement of traffic shaping and max bw controller only.

[..]

Quoted 18 lines
Quoted 9 lines
Quoted 8 lines
Quoted 2 lines

Can you please give little more details here regarding how QoS requirements are not met with proportional weight?

With proportional weights the whole bandwidth is allocated if no one else is using it. When IO is submitted other tasks with a higher weight can be forced to sleep until the IO generated by the low weight tasks is not completely dispatched. Or any extent of the priority inversion problems.

Quoted 8 lines

Hmm..., I am not very sure here. When admin is allocating the weights, he has the whole picture. He knows how many groups are conteding for the disk and what could be the worst case scenario. So if I have got two groups with A and B with weight 1 and 2 and both are contending, then as an admin one would expect to get 33% of BW for group A in worst case (if group B is continuously backlogged). If B is not contending than A can get 100% of BW. So while configuring the system, will one not plan for worst case (33% for A, and 66 % for B)?

Quoted 7 lines

OK, I'm quite convinced.. :)

To a large degree, if we want to provide a BW reservation strategy we must provide an interface that allows cgroups to ask for time slices such as max/min 5 IO requests every 50ms or something like that. Probably the same functionality can be achieved translating time slices from weights, percentages or absolute BW limits.

Ok, I would like to split it in two parts.

I think providng minimum gurantee in absolute terms like 5 IO request every 50ms will be very hard because IO scheduler has no control over how many competitors are there. An easier thing will be to have minimum gurantees on share basis. For minimum BW (disk time slice) gurantee, admin shall have to create right cgroup hierarchy and assign weights properly and then admin can calculate what % of disk slice a particular group will get as minimum gurantee. (This is more complicated than this as there are time slices which are not accounted to any groups. During queue switch cfq starts the time slice counting only after first request has completed to offset the impact of seeking and i guess also NCQ).

I think it should be possible to give max bandwidth gurantees in absolute terms, like io/s or sectors/sec or MB/sec etc, because only thing IO scheduler has to do is to not allow dispatch from a particular queue if it has crossed its limit and then either let the disk idle or move onto next eligible queue.

The only issue here will be async writes. max bw gurantee for async writes at IO scheduler level might not mean much to application because of page cache.

Quoted 8 lines
Quoted 4 lines

I see so much of the memory controller coming up. Since we've been discussing so many of these design points on mail, I wonder if it makes sense to summarize them somewhere (a wiki?). Would anyone like to take a shot at it?

Balbir, this is definitely a good idea. Just that once we have had some more discussion and some sort of understanding of issues, it might make more sense.

Sounds good. A wiki would be perfect IMHO, we could all contribute in the documentation, integrate thoughts, ideas and easily keep everything updated.

Quoted 4 lines

Got a question for you. Does memory controller already have the per cgroup dirty pages limit? If no, has this been discussed in the past? if yes, what was the conclsion?

I think the answer is in the previous email. :)

-Andrea

#52 lore raw

Re: IO controller discussion (Was: Re: [PATCH 01/10] Documentation)

In reply to Vivek Goyal
Quoted 131 lines · Vivek Goyal
Quoted 130 lines · Balbir Singh
Quoted 128 lines · Vivek Goyal
Quoted 49 lines · Andrea Righi
Quoted 9 lines · Vivek Goyal
Quoted 6 lines

I think it would be possible to implement both proportional and limiting rules at the same level (e.g., the IO scheduler), but we need also to address the memory consumption problem (I still need to review your patchset in details and I'm going to test it soon :), so I don't know if you already addressed this issue).

Can you please elaborate a bit on this? Are you concerned about that data structures created to solve the problem consume a lot of memory?

Sorry I was not very clear here. With memory consumption I mean wasting the memory with hard/slow reclaimable dirty pages or pending IO requests.

If there's only a global limit on dirty pages, any cgroup can exhaust that limit and cause other cgroups/processes to block when they try to write to disk.

But, ok, the IO controller is not probably the best place to implement such functionality. I should rework on the per cgroup dirty_ratio:

https://lists.linux-foundation.org/pipermail/containers/2008-September/013140.html

Last time we focused too much on the best interfaces to define dirty pages limit, and I never re-posted an updated version of this patchset. Now I think we can simply provide the same dirty_ratio/dirty_bytes interface that we provide globally, but per cgroup.

Quoted 20 lines
Quoted 19 lines

IOW if we simply don't dispatch requests and we don't throttle the tasks in the cgroup that exceeds its limit, how do we avoid the waste of memory due to the succeeding IO requests and the increasingly dirty pages in the page cache (that are also hard to reclaim)? I may be wrong, but I think we talked about this problem in a previous email... sorry I don't find the discussion in my mail archives.

IMHO a nice approach would be to measure IO consumption at the IO scheduler level, and control IO applying proportional weights / absolute limits _both_ at the IO scheduler / elevator level _and_ at the same time block the tasks from dirtying memory that will generate additional IO requests.

Anyway, there's no need to provide this with a single IO controller, we could split the problem in two parts: 1) provide a proportional / absolute IO controller in the IO schedulers and 2) allow to set, for example, a maximum limit of dirty pages for each cgroup.

Quoted 4 lines
Quoted 2 lines

I think setting a maximum limit on dirty pages is an interesting thought. It sounds like as if memory controller can handle it?

Exactly, the same above.

Thinking more about it. Memory controller can probably enforce the higher limit but it would not easily translate into a fixed upper async write rate. Till the process hits the page cache limit or is slowed down by dirty page writeout, it can get a very high async write BW.

So memory controller page cache limit will help but it would not direclty translate into what max bw limit patches are doing.

Even if we do max bw control at IO scheduler level, async writes are problematic again. IO controller will not be able to throttle the process until it sees actuall write request. In big memory systems, writeout might not happen for some time and till then it will see a high throughput.

So doing async write throttling at higher layer and not at IO scheduler layer gives us the opprotunity to produce more accurate results.

For sync requests, I think IO scheduler max bw control should work fine.

BTW, andrea, what is the use case of your patches? Andrew had mentioned that some people are already using it. I am curious to know will a proportional BW controller will solve the issues/requirements of these people or they have specific requirement of traffic shaping and max bw controller only.

[..]

Quoted 18 lines
Quoted 9 lines
Quoted 8 lines
Quoted 2 lines

Can you please give little more details here regarding how QoS requirements are not met with proportional weight?

With proportional weights the whole bandwidth is allocated if no one else is using it. When IO is submitted other tasks with a higher weight can be forced to sleep until the IO generated by the low weight tasks is not completely dispatched. Or any extent of the priority inversion problems.

Quoted 8 lines

Hmm..., I am not very sure here. When admin is allocating the weights, he has the whole picture. He knows how many groups are conteding for the disk and what could be the worst case scenario. So if I have got two groups with A and B with weight 1 and 2 and both are contending, then as an admin one would expect to get 33% of BW for group A in worst case (if group B is continuously backlogged). If B is not contending than A can get 100% of BW. So while configuring the system, will one not plan for worst case (33% for A, and 66 % for B)?

Quoted 7 lines

OK, I'm quite convinced.. :)

To a large degree, if we want to provide a BW reservation strategy we must provide an interface that allows cgroups to ask for time slices such as max/min 5 IO requests every 50ms or something like that. Probably the same functionality can be achieved translating time slices from weights, percentages or absolute BW limits.

Ok, I would like to split it in two parts.

I think providng minimum gurantee in absolute terms like 5 IO request every 50ms will be very hard because IO scheduler has no control over how many competitors are there. An easier thing will be to have minimum gurantees on share basis. For minimum BW (disk time slice) gurantee, admin shall have to create right cgroup hierarchy and assign weights properly and then admin can calculate what % of disk slice a particular group will get as minimum gurantee. (This is more complicated than this as there are time slices which are not accounted to any groups. During queue switch cfq starts the time slice counting only after first request has completed to offset the impact of seeking and i guess also NCQ).

I think it should be possible to give max bandwidth gurantees in absolute terms, like io/s or sectors/sec or MB/sec etc, because only thing IO scheduler has to do is to not allow dispatch from a particular queue if it has crossed its limit and then either let the disk idle or move onto next eligible queue.

The only issue here will be async writes. max bw gurantee for async writes at IO scheduler level might not mean much to application because of page cache.

Quoted 8 lines
Quoted 4 lines

I see so much of the memory controller coming up. Since we've been discussing so many of these design points on mail, I wonder if it makes sense to summarize them somewhere (a wiki?). Would anyone like to take a shot at it?

Balbir, this is definitely a good idea. Just that once we have had some more discussion and some sort of understanding of issues, it might make more sense.

Sounds good. A wiki would be perfect IMHO, we could all contribute in the documentation, integrate thoughts, ideas and easily keep everything updated.

Quoted 4 lines

Got a question for you. Does memory controller already have the per cgroup dirty pages limit? If no, has this been discussed in the past? if yes, what was the conclsion?

I think the answer is in the previous email. :)

-Andrea

#53 lore raw

Re: IO controller discussion (Was: Re: [PATCH 01/10] Documentation)

Quoted 5 lines · On Sun, 19 Apr 2009 17:53:59 +0200 Andrea Righi…
Quoted 4 lines

Got a question for you. Does memory controller already have the per cgroup dirty pages limit? If no, has this been discussed in the past? if yes, what was the conclsion?

IMHO, dirty page handling and I/O throttling is a different problem.

- A task (or cgroup) which makes the page dirty and
- A task (or cgroup) to which a page is accounted

Is different from each other, in general.

I have a plan to add dirty_ratio to memcg, but it's for avoiding massive stavation in memory reclaim, not for I/O control.

If you want to implement I/O throttle in MM layer, plz don't depend on memcg. The perpose is different.

Thanks,
-Kame

#54 lore raw

Re: IO controller discussion (Was: Re: [PATCH 01/10] Documentation)

In reply to Andrea Righi
Quoted 5 lines · On Sun, 19 Apr 2009 17:53:59 +0200 Andrea Righi <[email protected]> wrote:
Quoted 4 lines

Got a question for you. Does memory controller already have the per cgroup dirty pages limit? If no, has this been discussed in the past? if yes, what was the conclsion?

IMHO, dirty page handling and I/O throttling is a different problem.

- A task (or cgroup) which makes the page dirty and
- A task (or cgroup) to which a page is accounted

Is different from each other, in general.

I have a plan to add dirty_ratio to memcg, but it's for avoiding massive stavation in memory reclaim, not for I/O control.

If you want to implement I/O throttle in MM layer, plz don't depend on memcg. The perpose is different.

Thanks,
-Kame

#55 lore raw

Re: IO controller discussion (Was: Re: [PATCH 01/10] Documentation)

In reply to Vivek Goyal
Quoted 108 lines · Vivek Goyal
Quoted 49 lines · Andrea Righi
Quoted 9 lines · Vivek Goyal
Quoted 6 lines

I think it would be possible to implement both proportional and limiting rules at the same level (e.g., the IO scheduler), but we need also to address the memory consumption problem (I still need to review your patchset in details and I'm going to test it soon :), so I don't know if you already addressed this issue).

Can you please elaborate a bit on this? Are you concerned about that data structures created to solve the problem consume a lot of memory?

Sorry I was not very clear here. With memory consumption I mean wasting the memory with hard/slow reclaimable dirty pages or pending IO requests.

If there's only a global limit on dirty pages, any cgroup can exhaust that limit and cause other cgroups/processes to block when they try to write to disk.

But, ok, the IO controller is not probably the best place to implement such functionality. I should rework on the per cgroup dirty_ratio:

https://lists.linux-foundation.org/pipermail/containers/2008-September/013140.html

Last time we focused too much on the best interfaces to define dirty pages limit, and I never re-posted an updated version of this patchset. Now I think we can simply provide the same dirty_ratio/dirty_bytes interface that we provide globally, but per cgroup.

Quoted 20 lines
Quoted 19 lines

IOW if we simply don't dispatch requests and we don't throttle the tasks in the cgroup that exceeds its limit, how do we avoid the waste of memory due to the succeeding IO requests and the increasingly dirty pages in the page cache (that are also hard to reclaim)? I may be wrong, but I think we talked about this problem in a previous email... sorry I don't find the discussion in my mail archives.

IMHO a nice approach would be to measure IO consumption at the IO scheduler level, and control IO applying proportional weights / absolute limits _both_ at the IO scheduler / elevator level _and_ at the same time block the tasks from dirtying memory that will generate additional IO requests.

Anyway, there's no need to provide this with a single IO controller, we could split the problem in two parts: 1) provide a proportional / absolute IO controller in the IO schedulers and 2) allow to set, for example, a maximum limit of dirty pages for each cgroup.

Quoted 4 lines
Quoted 2 lines

I think setting a maximum limit on dirty pages is an interesting thought. It sounds like as if memory controller can handle it?

Exactly, the same above.

Thinking more about it. Memory controller can probably enforce the higher limit but it would not easily translate into a fixed upper async write rate. Till the process hits the page cache limit or is slowed down by dirty page writeout, it can get a very high async write BW.

So memory controller page cache limit will help but it would not direclty translate into what max bw limit patches are doing.

Even if we do max bw control at IO scheduler level, async writes are problematic again. IO controller will not be able to throttle the process until it sees actuall write request. In big memory systems, writeout might not happen for some time and till then it will see a high throughput.

So doing async write throttling at higher layer and not at IO scheduler layer gives us the opprotunity to produce more accurate results.

For sync requests, I think IO scheduler max bw control should work fine.

BTW, andrea, what is the use case of your patches? Andrew had mentioned that some people are already using it. I am curious to know will a proportional BW controller will solve the issues/requirements of these people or they have specific requirement of traffic shaping and max bw controller only.

[..]

Quoted 18 lines
Quoted 9 lines
Quoted 8 lines
Quoted 2 lines

Can you please give little more details here regarding how QoS requirements are not met with proportional weight?

With proportional weights the whole bandwidth is allocated if no one else is using it. When IO is submitted other tasks with a higher weight can be forced to sleep until the IO generated by the low weight tasks is not completely dispatched. Or any extent of the priority inversion problems.

Quoted 8 lines

Hmm..., I am not very sure here. When admin is allocating the weights, he has the whole picture. He knows how many groups are conteding for the disk and what could be the worst case scenario. So if I have got two groups with A and B with weight 1 and 2 and both are contending, then as an admin one would expect to get 33% of BW for group A in worst case (if group B is continuously backlogged). If B is not contending than A can get 100% of BW. So while configuring the system, will one not plan for worst case (33% for A, and 66 % for B)?

Quoted 7 lines

OK, I'm quite convinced.. :)

To a large degree, if we want to provide a BW reservation strategy we must provide an interface that allows cgroups to ask for time slices such as max/min 5 IO requests every 50ms or something like that. Probably the same functionality can be achieved translating time slices from weights, percentages or absolute BW limits.

Ok, I would like to split it in two parts.

Quoted 10 lines

I think providng minimum gurantee in absolute terms like 5 IO request every 50ms will be very hard because IO scheduler has no control over how many competitors are there. An easier thing will be to have minimum gurantees on share basis. For minimum BW (disk time slice) gurantee, admin shall have to create right cgroup hierarchy and assign weights properly and then admin can calculate what % of disk slice a particular group will get as minimum gurantee. (This is more complicated than this as there are time slices which are not accounted to any groups. During queue switch cfq starts the time slice counting only after first request has completed to offset the impact of seeking and i guess also NCQ).

I agree with Vivek that absolute metrics like 5 IO requests every 50ms might be hard to offer. But 'x ms of disk time every y ms, for a given cgroup' might be a desirable goal. That said, for now we can focus on weight based allocation of disk time, and leave such goals for future.

Quoted 14 lines

I think it should be possible to give max bandwidth gurantees in absolute terms, like io/s or sectors/sec or MB/sec etc, because only thing IO scheduler has to do is to not allow dispatch from a particular queue if it has crossed its limit and then either let the disk idle or move onto next eligible queue.

The only issue here will be async writes. max bw gurantee for async writes at IO scheduler level might not mean much to application because of page cache.

Thanks
Vivek

#56 lore raw

Re: [PATCH 01/10] Documentation

In reply to Vivek Goyal

Don't forget to update the 00-INDEX file when you add a new doc.^!^

Quoted 235 lines · Vivek Goyal

o Documentation for io-controller.

---
 Documentation/block/io-controller.txt |  221 +++++++++++++++++++++++++++++++++
 1 files changed, 221 insertions(+), 0 deletions(-)
 create mode 100644 Documentation/block/io-controller.txt
Documentation/block/io-controller.txt new file +221 -0
new file mode 100644
index 0000000..8884c5a
@@ -0,0 +1,221 @@
+				IO Controller
+				=============
+
+Overview
+========
+
+This patchset implements a proportional weight IO controller. That is one
+can create cgroups and assign prio/weights to those cgroups and task group
+will get access to disk proportionate to the weight of the group.
+
+These patches modify elevator layer and individual IO schedulers to do
+IO control hence this io controller works only on block devices which use
+one of the standard io schedulers can not be used with any xyz logical block
+device.
+
+The assumption/thought behind modifying IO scheduler is that resource control
+is needed only on leaf nodes where the actual contention for resources is
+present and not on intertermediate logical block devices.
+
+Consider following hypothetical scenario. Lets say there are three physical
+disks, namely sda, sdb and sdc. Two logical volumes (lv0 and lv1) have been
+created on top of these. Some part of sdb is in lv0 and some part is in lv1.
+
+			    lv0      lv1
+			  /	\  /     \
+			sda      sdb      sdc
+
+Also consider following cgroup hierarchy
+
+				root
+				/   \
+			       A     B
+			      / \    / \
+			     T1 T2  T3  T4
+
+A and B are two cgroups and T1, T2, T3 and T4 are tasks with-in those cgroups.
+Assuming T1, T2, T3 and T4 are doing IO on lv0 and lv1. These tasks should
+get their fair share of bandwidth on disks sda, sdb and sdc. There is no
+IO control on intermediate logical block nodes (lv0, lv1).
+
+So if tasks T1 and T2 are doing IO on lv0 and T3 and T4 are doing IO on lv1
+only, there will not be any contetion for resources between group A and B if
+IO is going to sda or sdc. But if actual IO gets translated to disk sdb, then
+IO scheduler associated with the sdb will distribute disk bandwidth to
+group A and B proportionate to their weight.
+
+CFQ already has the notion of fairness and it provides differential disk
+access based on priority and class of the task. Just that it is flat and
+with cgroup stuff, it needs to be made hierarchical.
+
+Rest of the IO schedulers (noop, deadline and AS) don't have any notion
+of fairness among various threads.
+
+One of the concerns raised with modifying IO schedulers was that we don't
+want to replicate the code in all the IO schedulers. These patches share
+the fair queuing code which has been moved to a common layer (elevator
+layer). Hence we don't end up replicating code across IO schedulers.
+
+Design
+======
+This patchset primarily uses BFQ (Budget Fair Queuing) code to provide
+fairness among different IO queues. Fabio and Paolo implemented BFQ which uses
+B-WF2Q+ algorithm for fair queuing.
+
+Why BFQ?
+
+- Not sure if weighted round robin logic of CFQ can be easily extended for
+  hierarchical mode. One of the things is that we can not keep dividing
+  the time slice of parent group among childrens. Deeper we go in hierarchy
+  time slice will get smaller.
+
+  One of the ways to implement hierarchical support could be to keep track
+  of virtual time and service provided to queue/group and select a queue/group
+  for service based on any of the various available algoriths.
+
+  BFQ already had support for hierarchical scheduling, taking those patches
+  was easier.
+
+- BFQ was designed to provide tighter bounds/delay w.r.t service provided
+  to a queue. Delay/Jitter with BFQ is supposed to be O(1).
+
+  Note: BFQ originally used amount of IO done (number of sectors) as notion
+        of service provided. IOW, it tried to provide fairness in terms of
+        actual IO done and not in terms of actual time disk access was
+	given to a queue.
+
+	This patcheset modified BFQ to provide fairness in time domain because
+	that's what CFQ does. So idea was try not to deviate too much from
+	the CFQ behavior initially.
+
+	Providing fairness in time domain makes accounting trciky because
+	due to command queueing, at one time there might be multiple requests
+	from different queues and there is no easy way to find out how much
+	disk time actually was consumed by the requests of a particular
+	queue. More about this in comments in source code.
+
+So it is yet to be seen if changing to time domain still retains BFQ gurantees
+or not.
+
+From data structure point of view, one can think of a tree per device, where
+io groups and io queues are hanging and are being scheduled using B-WF2Q+
+algorithm. io_queue, is end queue where requests are actually stored and
+dispatched from (like cfqq).
+
+These io queues are primarily created by and managed by end io schedulers
+depending on its semantics. For example, noop, deadline and AS ioschedulers
+keep one io queues per cgroup and cfqq keeps one io queue per io_context in
+a cgroup (apart from async queues).
+
+A request is mapped to an io group by elevator layer and which io queue it
+is mapped to with in group depends on ioscheduler. Currently "current" task
+is used to determine the cgroup (hence io group) of the request. Down the
+line we need to make use of bio-cgroup patches to map delayed writes to
+right group.
+
+Going back to old behavior
+==========================
+In new scheme of things essentially we are creating hierarchical fair
+queuing logic in elevator layer and chaning IO schedulers to make use of
+that logic so that end IO schedulers start supporting hierarchical scheduling.
+
+Elevator layer continues to support the old interfaces. So even if fair queuing
+is enabled at elevator layer, one can have both new hierchical scheduler as
+well as old non-hierarchical scheduler operating.
+
+Also noop, deadline and AS have option of enabling hierarchical scheduling.
+If it is selected, fair queuing is done in hierarchical manner. If hierarchical
+scheduling is disabled, noop, deadline and AS should retain their existing
+behavior.
+
+CFQ is the only exception where one can not disable fair queuing as it is
+needed for provding fairness among various threads even in non-hierarchical
+mode.
+
+Various user visible config options
+===================================
+CONFIG_IOSCHED_NOOP_HIER
+	- Enables hierchical fair queuing in noop. Not selecting this option
+	  leads to old behavior of noop.
+
+CONFIG_IOSCHED_DEADLINE_HIER
+	- Enables hierchical fair queuing in deadline. Not selecting this
+	  option leads to old behavior of deadline.
+
+CONFIG_IOSCHED_AS_HIER
+	- Enables hierchical fair queuing in AS. Not selecting this option
+	  leads to old behavior of AS.
+
+CONFIG_IOSCHED_CFQ_HIER
+	- Enables hierarchical fair queuing in CFQ. Not selecting this option
+	  still does fair queuing among various queus but it is flat and not
+	  hierarchical.
+
+Config options selected automatically
+=====================================
+These config options are not user visible and are selected/deselected
+automatically based on IO scheduler configurations.
+
+CONFIG_ELV_FAIR_QUEUING
+	- Enables/Disables the fair queuing logic at elevator layer.
+
+CONFIG_GROUP_IOSCHED
+	- Enables/Disables hierarchical queuing and associated cgroup bits.
+
+TODO
+====
+- Lots of cleanups, testing, bug fixing, optimizations, benchmarking etc...
+- Convert cgroup ioprio to notion of weight.
+- Anticipatory code will need more work. It is not working properly currently
+  and needs more thought.
+- Use of bio-cgroup patches.
+- Use of Nauman's per cgroup request descriptor patches.
+
+HOWTO
+=====
+So far I have done very simple testing of running two dd threads in two
+different cgroups. Here is what you can do.
+
+- Enable hierarchical scheduling in io scheuduler of your choice (say cfq).
+	CONFIG_IOSCHED_CFQ_HIER=y
+
+- Compile and boot into kernel and mount IO controller.
+
+	mount -t cgroup -o io none /cgroup
+
+- Create two cgroups
+	mkdir -p /cgroup/test1/ /cgroup/test2
+
+- Set io priority of group test1 and test2
+	echo 0 > /cgroup/test1/io.ioprio
+	echo 4 > /cgroup/test2/io.ioprio
+
+- Create two same size files (say 512MB each) on same disk (file1, file2) and
+  launch two dd threads in different cgroup to read those files. Make sure
+  right io scheduler is being used for the block device where files are
+  present (the one you compiled in hierarchical mode).
+
+	echo 1 > /proc/sys/vm/drop_caches
+
+	dd if=/mnt/lv0/zerofile1 of=/dev/null &
+	echo $! > /cgroup/test1/tasks
+	cat /cgroup/test1/tasks
+
+	dd if=/mnt/lv0/zerofile2 of=/dev/null &
+	echo $! > /cgroup/test2/tasks
+	cat /cgroup/test2/tasks
+
+- First dd should finish first.
+
+Some Test Results
+=================
+- Two dd in two cgroups with prio 0 and 4. Ran two "dd" in those cgroups.
+
+234179072 bytes (234 MB) copied, 10.1811 s, 23.0 MB/s
+234179072 bytes (234 MB) copied, 12.6187 s, 18.6 MB/s
+
+- Three dd in three cgroups with prio 0, 4, 4.
+
+234179072 bytes (234 MB) copied, 13.7654 s, 17.0 MB/s
+234179072 bytes (234 MB) copied, 19.476 s, 12.0 MB/s
+234179072 bytes (234 MB) copied, 20.1858 s, 11.6 MB/s
Signature · 2 lines
Regards
Yang Hongyang

A message that is not in the archive (sent off-list or not yet received): [email protected]

#58 lore raw

Re: [PATCH 01/10] Documentation

Quoted 2 lines · Yang Hongyang

Don't forget to update the 00-INDEX file when you add a new doc.^!^

Thanks. Will do it.

Vivek

Quoted 241 lines
Quoted 235 lines · Vivek Goyal

o Documentation for io-controller.

---
 Documentation/block/io-controller.txt |  221 +++++++++++++++++++++++++++++++++
 1 files changed, 221 insertions(+), 0 deletions(-)
 create mode 100644 Documentation/block/io-controller.txt
Documentation/block/io-controller.txt new file +221 -0
new file mode 100644
index 0000000..8884c5a
@@ -0,0 +1,221 @@
+				IO Controller
+				=============
+
+Overview
+========
+
+This patchset implements a proportional weight IO controller. That is one
+can create cgroups and assign prio/weights to those cgroups and task group
+will get access to disk proportionate to the weight of the group.
+
+These patches modify elevator layer and individual IO schedulers to do
+IO control hence this io controller works only on block devices which use
+one of the standard io schedulers can not be used with any xyz logical block
+device.
+
+The assumption/thought behind modifying IO scheduler is that resource control
+is needed only on leaf nodes where the actual contention for resources is
+present and not on intertermediate logical block devices.
+
+Consider following hypothetical scenario. Lets say there are three physical
+disks, namely sda, sdb and sdc. Two logical volumes (lv0 and lv1) have been
+created on top of these. Some part of sdb is in lv0 and some part is in lv1.
+
+			    lv0      lv1
+			  /	\  /     \
+			sda      sdb      sdc
+
+Also consider following cgroup hierarchy
+
+				root
+				/   \
+			       A     B
+			      / \    / \
+			     T1 T2  T3  T4
+
+A and B are two cgroups and T1, T2, T3 and T4 are tasks with-in those cgroups.
+Assuming T1, T2, T3 and T4 are doing IO on lv0 and lv1. These tasks should
+get their fair share of bandwidth on disks sda, sdb and sdc. There is no
+IO control on intermediate logical block nodes (lv0, lv1).
+
+So if tasks T1 and T2 are doing IO on lv0 and T3 and T4 are doing IO on lv1
+only, there will not be any contetion for resources between group A and B if
+IO is going to sda or sdc. But if actual IO gets translated to disk sdb, then
+IO scheduler associated with the sdb will distribute disk bandwidth to
+group A and B proportionate to their weight.
+
+CFQ already has the notion of fairness and it provides differential disk
+access based on priority and class of the task. Just that it is flat and
+with cgroup stuff, it needs to be made hierarchical.
+
+Rest of the IO schedulers (noop, deadline and AS) don't have any notion
+of fairness among various threads.
+
+One of the concerns raised with modifying IO schedulers was that we don't
+want to replicate the code in all the IO schedulers. These patches share
+the fair queuing code which has been moved to a common layer (elevator
+layer). Hence we don't end up replicating code across IO schedulers.
+
+Design
+======
+This patchset primarily uses BFQ (Budget Fair Queuing) code to provide
+fairness among different IO queues. Fabio and Paolo implemented BFQ which uses
+B-WF2Q+ algorithm for fair queuing.
+
+Why BFQ?
+
+- Not sure if weighted round robin logic of CFQ can be easily extended for
+  hierarchical mode. One of the things is that we can not keep dividing
+  the time slice of parent group among childrens. Deeper we go in hierarchy
+  time slice will get smaller.
+
+  One of the ways to implement hierarchical support could be to keep track
+  of virtual time and service provided to queue/group and select a queue/group
+  for service based on any of the various available algoriths.
+
+  BFQ already had support for hierarchical scheduling, taking those patches
+  was easier.
+
+- BFQ was designed to provide tighter bounds/delay w.r.t service provided
+  to a queue. Delay/Jitter with BFQ is supposed to be O(1).
+
+  Note: BFQ originally used amount of IO done (number of sectors) as notion
+        of service provided. IOW, it tried to provide fairness in terms of
+        actual IO done and not in terms of actual time disk access was
+	given to a queue.
+
+	This patcheset modified BFQ to provide fairness in time domain because
+	that's what CFQ does. So idea was try not to deviate too much from
+	the CFQ behavior initially.
+
+	Providing fairness in time domain makes accounting trciky because
+	due to command queueing, at one time there might be multiple requests
+	from different queues and there is no easy way to find out how much
+	disk time actually was consumed by the requests of a particular
+	queue. More about this in comments in source code.
+
+So it is yet to be seen if changing to time domain still retains BFQ gurantees
+or not.
+
+From data structure point of view, one can think of a tree per device, where
+io groups and io queues are hanging and are being scheduled using B-WF2Q+
+algorithm. io_queue, is end queue where requests are actually stored and
+dispatched from (like cfqq).
+
+These io queues are primarily created by and managed by end io schedulers
+depending on its semantics. For example, noop, deadline and AS ioschedulers
+keep one io queues per cgroup and cfqq keeps one io queue per io_context in
+a cgroup (apart from async queues).
+
+A request is mapped to an io group by elevator layer and which io queue it
+is mapped to with in group depends on ioscheduler. Currently "current" task
+is used to determine the cgroup (hence io group) of the request. Down the
+line we need to make use of bio-cgroup patches to map delayed writes to
+right group.
+
+Going back to old behavior
+==========================
+In new scheme of things essentially we are creating hierarchical fair
+queuing logic in elevator layer and chaning IO schedulers to make use of
+that logic so that end IO schedulers start supporting hierarchical scheduling.
+
+Elevator layer continues to support the old interfaces. So even if fair queuing
+is enabled at elevator layer, one can have both new hierchical scheduler as
+well as old non-hierarchical scheduler operating.
+
+Also noop, deadline and AS have option of enabling hierarchical scheduling.
+If it is selected, fair queuing is done in hierarchical manner. If hierarchical
+scheduling is disabled, noop, deadline and AS should retain their existing
+behavior.
+
+CFQ is the only exception where one can not disable fair queuing as it is
+needed for provding fairness among various threads even in non-hierarchical
+mode.
+
+Various user visible config options
+===================================
+CONFIG_IOSCHED_NOOP_HIER
+	- Enables hierchical fair queuing in noop. Not selecting this option
+	  leads to old behavior of noop.
+
+CONFIG_IOSCHED_DEADLINE_HIER
+	- Enables hierchical fair queuing in deadline. Not selecting this
+	  option leads to old behavior of deadline.
+
+CONFIG_IOSCHED_AS_HIER
+	- Enables hierchical fair queuing in AS. Not selecting this option
+	  leads to old behavior of AS.
+
+CONFIG_IOSCHED_CFQ_HIER
+	- Enables hierarchical fair queuing in CFQ. Not selecting this option
+	  still does fair queuing among various queus but it is flat and not
+	  hierarchical.
+
+Config options selected automatically
+=====================================
+These config options are not user visible and are selected/deselected
+automatically based on IO scheduler configurations.
+
+CONFIG_ELV_FAIR_QUEUING
+	- Enables/Disables the fair queuing logic at elevator layer.
+
+CONFIG_GROUP_IOSCHED
+	- Enables/Disables hierarchical queuing and associated cgroup bits.
+
+TODO
+====
+- Lots of cleanups, testing, bug fixing, optimizations, benchmarking etc...
+- Convert cgroup ioprio to notion of weight.
+- Anticipatory code will need more work. It is not working properly currently
+  and needs more thought.
+- Use of bio-cgroup patches.
+- Use of Nauman's per cgroup request descriptor patches.
+
+HOWTO
+=====
+So far I have done very simple testing of running two dd threads in two
+different cgroups. Here is what you can do.
+
+- Enable hierarchical scheduling in io scheuduler of your choice (say cfq).
+	CONFIG_IOSCHED_CFQ_HIER=y
+
+- Compile and boot into kernel and mount IO controller.
+
+	mount -t cgroup -o io none /cgroup
+
+- Create two cgroups
+	mkdir -p /cgroup/test1/ /cgroup/test2
+
+- Set io priority of group test1 and test2
+	echo 0 > /cgroup/test1/io.ioprio
+	echo 4 > /cgroup/test2/io.ioprio
+
+- Create two same size files (say 512MB each) on same disk (file1, file2) and
+  launch two dd threads in different cgroup to read those files. Make sure
+  right io scheduler is being used for the block device where files are
+  present (the one you compiled in hierarchical mode).
+
+	echo 1 > /proc/sys/vm/drop_caches
+
+	dd if=/mnt/lv0/zerofile1 of=/dev/null &
+	echo $! > /cgroup/test1/tasks
+	cat /cgroup/test1/tasks
+
+	dd if=/mnt/lv0/zerofile2 of=/dev/null &
+	echo $! > /cgroup/test2/tasks
+	cat /cgroup/test2/tasks
+
+- First dd should finish first.
+
+Some Test Results
+=================
+- Two dd in two cgroups with prio 0 and 4. Ran two "dd" in those cgroups.
+
+234179072 bytes (234 MB) copied, 10.1811 s, 23.0 MB/s
+234179072 bytes (234 MB) copied, 12.6187 s, 18.6 MB/s
+
+- Three dd in three cgroups with prio 0, 4, 4.
+
+234179072 bytes (234 MB) copied, 13.7654 s, 17.0 MB/s
+234179072 bytes (234 MB) copied, 19.476 s, 12.0 MB/s
+234179072 bytes (234 MB) copied, 20.1858 s, 11.6 MB/s
Signature · 2 lines
Regards
Yang Hongyang
#59 lore raw

Re: [PATCH 01/10] Documentation

In reply to Yang Hongyang
Quoted 2 lines · Yang Hongyang

Don't forget to update the 00-INDEX file when you add a new doc.^!^

Thanks. Will do it.

Vivek

Quoted 241 lines
Quoted 235 lines · Vivek Goyal

o Documentation for io-controller.

---
 Documentation/block/io-controller.txt |  221 +++++++++++++++++++++++++++++++++
 1 files changed, 221 insertions(+), 0 deletions(-)
 create mode 100644 Documentation/block/io-controller.txt
Documentation/block/io-controller.txt new file +221 -0
new file mode 100644
index 0000000..8884c5a
@@ -0,0 +1,221 @@
+				IO Controller
+				=============
+
+Overview
+========
+
+This patchset implements a proportional weight IO controller. That is one
+can create cgroups and assign prio/weights to those cgroups and task group
+will get access to disk proportionate to the weight of the group.
+
+These patches modify elevator layer and individual IO schedulers to do
+IO control hence this io controller works only on block devices which use
+one of the standard io schedulers can not be used with any xyz logical block
+device.
+
+The assumption/thought behind modifying IO scheduler is that resource control
+is needed only on leaf nodes where the actual contention for resources is
+present and not on intertermediate logical block devices.
+
+Consider following hypothetical scenario. Lets say there are three physical
+disks, namely sda, sdb and sdc. Two logical volumes (lv0 and lv1) have been
+created on top of these. Some part of sdb is in lv0 and some part is in lv1.
+
+			    lv0      lv1
+			  /	\  /     \
+			sda      sdb      sdc
+
+Also consider following cgroup hierarchy
+
+				root
+				/   \
+			       A     B
+			      / \    / \
+			     T1 T2  T3  T4
+
+A and B are two cgroups and T1, T2, T3 and T4 are tasks with-in those cgroups.
+Assuming T1, T2, T3 and T4 are doing IO on lv0 and lv1. These tasks should
+get their fair share of bandwidth on disks sda, sdb and sdc. There is no
+IO control on intermediate logical block nodes (lv0, lv1).
+
+So if tasks T1 and T2 are doing IO on lv0 and T3 and T4 are doing IO on lv1
+only, there will not be any contetion for resources between group A and B if
+IO is going to sda or sdc. But if actual IO gets translated to disk sdb, then
+IO scheduler associated with the sdb will distribute disk bandwidth to
+group A and B proportionate to their weight.
+
+CFQ already has the notion of fairness and it provides differential disk
+access based on priority and class of the task. Just that it is flat and
+with cgroup stuff, it needs to be made hierarchical.
+
+Rest of the IO schedulers (noop, deadline and AS) don't have any notion
+of fairness among various threads.
+
+One of the concerns raised with modifying IO schedulers was that we don't
+want to replicate the code in all the IO schedulers. These patches share
+the fair queuing code which has been moved to a common layer (elevator
+layer). Hence we don't end up replicating code across IO schedulers.
+
+Design
+======
+This patchset primarily uses BFQ (Budget Fair Queuing) code to provide
+fairness among different IO queues. Fabio and Paolo implemented BFQ which uses
+B-WF2Q+ algorithm for fair queuing.
+
+Why BFQ?
+
+- Not sure if weighted round robin logic of CFQ can be easily extended for
+  hierarchical mode. One of the things is that we can not keep dividing
+  the time slice of parent group among childrens. Deeper we go in hierarchy
+  time slice will get smaller.
+
+  One of the ways to implement hierarchical support could be to keep track
+  of virtual time and service provided to queue/group and select a queue/group
+  for service based on any of the various available algoriths.
+
+  BFQ already had support for hierarchical scheduling, taking those patches
+  was easier.
+
+- BFQ was designed to provide tighter bounds/delay w.r.t service provided
+  to a queue. Delay/Jitter with BFQ is supposed to be O(1).
+
+  Note: BFQ originally used amount of IO done (number of sectors) as notion
+        of service provided. IOW, it tried to provide fairness in terms of
+        actual IO done and not in terms of actual time disk access was
+	given to a queue.
+
+	This patcheset modified BFQ to provide fairness in time domain because
+	that's what CFQ does. So idea was try not to deviate too much from
+	the CFQ behavior initially.
+
+	Providing fairness in time domain makes accounting trciky because
+	due to command queueing, at one time there might be multiple requests
+	from different queues and there is no easy way to find out how much
+	disk time actually was consumed by the requests of a particular
+	queue. More about this in comments in source code.
+
+So it is yet to be seen if changing to time domain still retains BFQ gurantees
+or not.
+
+From data structure point of view, one can think of a tree per device, where
+io groups and io queues are hanging and are being scheduled using B-WF2Q+
+algorithm. io_queue, is end queue where requests are actually stored and
+dispatched from (like cfqq).
+
+These io queues are primarily created by and managed by end io schedulers
+depending on its semantics. For example, noop, deadline and AS ioschedulers
+keep one io queues per cgroup and cfqq keeps one io queue per io_context in
+a cgroup (apart from async queues).
+
+A request is mapped to an io group by elevator layer and which io queue it
+is mapped to with in group depends on ioscheduler. Currently "current" task
+is used to determine the cgroup (hence io group) of the request. Down the
+line we need to make use of bio-cgroup patches to map delayed writes to
+right group.
+
+Going back to old behavior
+==========================
+In new scheme of things essentially we are creating hierarchical fair
+queuing logic in elevator layer and chaning IO schedulers to make use of
+that logic so that end IO schedulers start supporting hierarchical scheduling.
+
+Elevator layer continues to support the old interfaces. So even if fair queuing
+is enabled at elevator layer, one can have both new hierchical scheduler as
+well as old non-hierarchical scheduler operating.
+
+Also noop, deadline and AS have option of enabling hierarchical scheduling.
+If it is selected, fair queuing is done in hierarchical manner. If hierarchical
+scheduling is disabled, noop, deadline and AS should retain their existing
+behavior.
+
+CFQ is the only exception where one can not disable fair queuing as it is
+needed for provding fairness among various threads even in non-hierarchical
+mode.
+
+Various user visible config options
+===================================
+CONFIG_IOSCHED_NOOP_HIER
+	- Enables hierchical fair queuing in noop. Not selecting this option
+	  leads to old behavior of noop.
+
+CONFIG_IOSCHED_DEADLINE_HIER
+	- Enables hierchical fair queuing in deadline. Not selecting this
+	  option leads to old behavior of deadline.
+
+CONFIG_IOSCHED_AS_HIER
+	- Enables hierchical fair queuing in AS. Not selecting this option
+	  leads to old behavior of AS.
+
+CONFIG_IOSCHED_CFQ_HIER
+	- Enables hierarchical fair queuing in CFQ. Not selecting this option
+	  still does fair queuing among various queus but it is flat and not
+	  hierarchical.
+
+Config options selected automatically
+=====================================
+These config options are not user visible and are selected/deselected
+automatically based on IO scheduler configurations.
+
+CONFIG_ELV_FAIR_QUEUING
+	- Enables/Disables the fair queuing logic at elevator layer.
+
+CONFIG_GROUP_IOSCHED
+	- Enables/Disables hierarchical queuing and associated cgroup bits.
+
+TODO
+====
+- Lots of cleanups, testing, bug fixing, optimizations, benchmarking etc...
+- Convert cgroup ioprio to notion of weight.
+- Anticipatory code will need more work. It is not working properly currently
+  and needs more thought.
+- Use of bio-cgroup patches.
+- Use of Nauman's per cgroup request descriptor patches.
+
+HOWTO
+=====
+So far I have done very simple testing of running two dd threads in two
+different cgroups. Here is what you can do.
+
+- Enable hierarchical scheduling in io scheuduler of your choice (say cfq).
+	CONFIG_IOSCHED_CFQ_HIER=y
+
+- Compile and boot into kernel and mount IO controller.
+
+	mount -t cgroup -o io none /cgroup
+
+- Create two cgroups
+	mkdir -p /cgroup/test1/ /cgroup/test2
+
+- Set io priority of group test1 and test2
+	echo 0 > /cgroup/test1/io.ioprio
+	echo 4 > /cgroup/test2/io.ioprio
+
+- Create two same size files (say 512MB each) on same disk (file1, file2) and
+  launch two dd threads in different cgroup to read those files. Make sure
+  right io scheduler is being used for the block device where files are
+  present (the one you compiled in hierarchical mode).
+
+	echo 1 > /proc/sys/vm/drop_caches
+
+	dd if=/mnt/lv0/zerofile1 of=/dev/null &
+	echo $! > /cgroup/test1/tasks
+	cat /cgroup/test1/tasks
+
+	dd if=/mnt/lv0/zerofile2 of=/dev/null &
+	echo $! > /cgroup/test2/tasks
+	cat /cgroup/test2/tasks
+
+- First dd should finish first.
+
+Some Test Results
+=================
+- Two dd in two cgroups with prio 0 and 4. Ran two "dd" in those cgroups.
+
+234179072 bytes (234 MB) copied, 10.1811 s, 23.0 MB/s
+234179072 bytes (234 MB) copied, 12.6187 s, 18.6 MB/s
+
+- Three dd in three cgroups with prio 0, 4, 4.
+
+234179072 bytes (234 MB) copied, 13.7654 s, 17.0 MB/s
+234179072 bytes (234 MB) copied, 19.476 s, 12.0 MB/s
+234179072 bytes (234 MB) copied, 20.1858 s, 11.6 MB/s
Signature · 2 lines
Regards
Yang Hongyang

A message that is not in the archive (sent off-list or not yet received): 1236823015-4183-2-git-send-email-vgoyal-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org

#61 lore raw

Re: [PATCH 01/10] Documentation

Quoted 225 lines · Vivek Goyal

o Documentation for io-controller.

---
 Documentation/block/io-controller.txt |  221 +++++++++++++++++++++++++++++++++
 1 files changed, 221 insertions(+), 0 deletions(-)
 create mode 100644 Documentation/block/io-controller.txt
Documentation/block/io-controller.txt new file +211 -0
new file mode 100644
index 0000000..8884c5a
@@ -0,0 +1,221 @@
+				IO Controller
+				=============
+
+Overview
+========
+
+This patchset implements a proportional weight IO controller. That is one
+can create cgroups and assign prio/weights to those cgroups and task group
+will get access to disk proportionate to the weight of the group.
+
+These patches modify elevator layer and individual IO schedulers to do
+IO control hence this io controller works only on block devices which use
+one of the standard io schedulers can not be used with any xyz logical block
+device.
+
+The assumption/thought behind modifying IO scheduler is that resource control
+is needed only on leaf nodes where the actual contention for resources is
+present and not on intertermediate logical block devices.
+
+Consider following hypothetical scenario. Lets say there are three physical
+disks, namely sda, sdb and sdc. Two logical volumes (lv0 and lv1) have been
+created on top of these. Some part of sdb is in lv0 and some part is in lv1.
+
+			    lv0      lv1
+			  /	\  /     \
+			sda      sdb      sdc
+
+Also consider following cgroup hierarchy
+
+				root
+				/   \
+			       A     B
+			      / \    / \
+			     T1 T2  T3  T4
+
+A and B are two cgroups and T1, T2, T3 and T4 are tasks with-in those cgroups.
+Assuming T1, T2, T3 and T4 are doing IO on lv0 and lv1. These tasks should
+get their fair share of bandwidth on disks sda, sdb and sdc. There is no
+IO control on intermediate logical block nodes (lv0, lv1).
+
+So if tasks T1 and T2 are doing IO on lv0 and T3 and T4 are doing IO on lv1
+only, there will not be any contetion for resources between group A and B if
+IO is going to sda or sdc. But if actual IO gets translated to disk sdb, then
+IO scheduler associated with the sdb will distribute disk bandwidth to
+group A and B proportionate to their weight.
+
+CFQ already has the notion of fairness and it provides differential disk
+access based on priority and class of the task. Just that it is flat and
+with cgroup stuff, it needs to be made hierarchical.
+
+Rest of the IO schedulers (noop, deadline and AS) don't have any notion
+of fairness among various threads.
+
+One of the concerns raised with modifying IO schedulers was that we don't
+want to replicate the code in all the IO schedulers. These patches share
+the fair queuing code which has been moved to a common layer (elevator
+layer). Hence we don't end up replicating code across IO schedulers.
+
+Design
+======
+This patchset primarily uses BFQ (Budget Fair Queuing) code to provide
+fairness among different IO queues. Fabio and Paolo implemented BFQ which uses
+B-WF2Q+ algorithm for fair queuing.
+
+Why BFQ?
+
+- Not sure if weighted round robin logic of CFQ can be easily extended for
+  hierarchical mode. One of the things is that we can not keep dividing
+  the time slice of parent group among childrens. Deeper we go in hierarchy
+  time slice will get smaller.
+
+  One of the ways to implement hierarchical support could be to keep track
+  of virtual time and service provided to queue/group and select a queue/group
+  for service based on any of the various available algoriths.
+
+  BFQ already had support for hierarchical scheduling, taking those patches
+  was easier.
+
+- BFQ was designed to provide tighter bounds/delay w.r.t service provided
+  to a queue. Delay/Jitter with BFQ is supposed to be O(1).
+
+  Note: BFQ originally used amount of IO done (number of sectors) as notion
+        of service provided. IOW, it tried to provide fairness in terms of
+        actual IO done and not in terms of actual time disk access was
+	given to a queue.
+
+	This patcheset modified BFQ to provide fairness in time domain because
+	that's what CFQ does. So idea was try not to deviate too much from
+	the CFQ behavior initially.
+
+	Providing fairness in time domain makes accounting trciky because
+	due to command queueing, at one time there might be multiple requests
+	from different queues and there is no easy way to find out how much
+	disk time actually was consumed by the requests of a particular
+	queue. More about this in comments in source code.
+
+So it is yet to be seen if changing to time domain still retains BFQ gurantees
+or not.
+
+From data structure point of view, one can think of a tree per device, where
+io groups and io queues are hanging and are being scheduled using B-WF2Q+
+algorithm. io_queue, is end queue where requests are actually stored and
+dispatched from (like cfqq).
+
+These io queues are primarily created by and managed by end io schedulers
+depending on its semantics. For example, noop, deadline and AS ioschedulers
+keep one io queues per cgroup and cfqq keeps one io queue per io_context in
+a cgroup (apart from async queues).
+
+A request is mapped to an io group by elevator layer and which io queue it
+is mapped to with in group depends on ioscheduler. Currently "current" task
+is used to determine the cgroup (hence io group) of the request. Down the
+line we need to make use of bio-cgroup patches to map delayed writes to
+right group.
+
+Going back to old behavior
+==========================
+In new scheme of things essentially we are creating hierarchical fair
+queuing logic in elevator layer and chaning IO schedulers to make use of
+that logic so that end IO schedulers start supporting hierarchical scheduling.
+
+Elevator layer continues to support the old interfaces. So even if fair queuing
+is enabled at elevator layer, one can have both new hierchical scheduler as
+well as old non-hierarchical scheduler operating.
+
+Also noop, deadline and AS have option of enabling hierarchical scheduling.
+If it is selected, fair queuing is done in hierarchical manner. If hierarchical
+scheduling is disabled, noop, deadline and AS should retain their existing
+behavior.
+
+CFQ is the only exception where one can not disable fair queuing as it is
+needed for provding fairness among various threads even in non-hierarchical
+mode.
+
+Various user visible config options
+===================================
+CONFIG_IOSCHED_NOOP_HIER
+	- Enables hierchical fair queuing in noop. Not selecting this option
+	  leads to old behavior of noop.
+
+CONFIG_IOSCHED_DEADLINE_HIER
+	- Enables hierchical fair queuing in deadline. Not selecting this
+	  option leads to old behavior of deadline.
+
+CONFIG_IOSCHED_AS_HIER
+	- Enables hierchical fair queuing in AS. Not selecting this option
+	  leads to old behavior of AS.
+
+CONFIG_IOSCHED_CFQ_HIER
+	- Enables hierarchical fair queuing in CFQ. Not selecting this option
+	  still does fair queuing among various queus but it is flat and not
+	  hierarchical.
+
+Config options selected automatically
+=====================================
+These config options are not user visible and are selected/deselected
+automatically based on IO scheduler configurations.
+
+CONFIG_ELV_FAIR_QUEUING
+	- Enables/Disables the fair queuing logic at elevator layer.
+
+CONFIG_GROUP_IOSCHED
+	- Enables/Disables hierarchical queuing and associated cgroup bits.
+
+TODO
+====
+- Lots of cleanups, testing, bug fixing, optimizations, benchmarking etc...
+- Convert cgroup ioprio to notion of weight.
+- Anticipatory code will need more work. It is not working properly currently
+  and needs more thought.
+- Use of bio-cgroup patches.
+- Use of Nauman's per cgroup request descriptor patches.
+
+HOWTO
+=====
+So far I have done very simple testing of running two dd threads in two
+different cgroups. Here is what you can do.
+
+- Enable hierarchical scheduling in io scheuduler of your choice (say cfq).
+	CONFIG_IOSCHED_CFQ_HIER=y
+
+- Compile and boot into kernel and mount IO controller.
+
+	mount -t cgroup -o io none /cgroup
+
+- Create two cgroups
+	mkdir -p /cgroup/test1/ /cgroup/test2
+
+- Set io priority of group test1 and test2
+	echo 0 > /cgroup/test1/io.ioprio
+	echo 4 > /cgroup/test2/io.ioprio
+
+- Create two same size files (say 512MB each) on same disk (file1, file2) and
+  launch two dd threads in different cgroup to read those files. Make sure
+  right io scheduler is being used for the block device where files are
+  present (the one you compiled in hierarchical mode).
+
+	echo 1 > /proc/sys/vm/drop_caches
+
+	dd if=/mnt/lv0/zerofile1 of=/dev/null &
+	echo $! > /cgroup/test1/tasks
+	cat /cgroup/test1/tasks
+
+	dd if=/mnt/lv0/zerofile2 of=/dev/null &
+	echo $! > /cgroup/test2/tasks
+	cat /cgroup/test2/tasks
+
+- First dd should finish first.
+
+Some Test Results
+=================

Documentation/block/io-controller.txt line 221

Quoted 10 lines
Documentation/block/io-controller.txt new file +10 -0
+- Two dd in two cgroups with prio 0 and 4. Ran two "dd" in those cgroups.
+
+234179072 bytes (234 MB) copied, 10.1811 s, 23.0 MB/s
+234179072 bytes (234 MB) copied, 12.6187 s, 18.6 MB/s
+
+- Three dd in three cgroups with prio 0, 4, 4.
+
+234179072 bytes (234 MB) copied, 13.7654 s, 17.0 MB/s
+234179072 bytes (234 MB) copied, 19.476 s, 12.0 MB/s
+234179072 bytes (234 MB) copied, 20.1858 s, 11.6 MB/s

Hi Vivek,

I would be interested in knowing if these are the results expected?

Signature · 2 lines
regards,
Dhaval
#62 lore raw

Re: [PATCH 01/10] Documentation

In reply to Vivek Goyal
Quoted 225 lines · Vivek Goyal

o Documentation for io-controller.

---
 Documentation/block/io-controller.txt |  221 +++++++++++++++++++++++++++++++++
 1 files changed, 221 insertions(+), 0 deletions(-)
 create mode 100644 Documentation/block/io-controller.txt
Documentation/block/io-controller.txt new file +211 -0
new file mode 100644
index 0000000..8884c5a
@@ -0,0 +1,221 @@
+				IO Controller
+				=============
+
+Overview
+========
+
+This patchset implements a proportional weight IO controller. That is one
+can create cgroups and assign prio/weights to those cgroups and task group
+will get access to disk proportionate to the weight of the group.
+
+These patches modify elevator layer and individual IO schedulers to do
+IO control hence this io controller works only on block devices which use
+one of the standard io schedulers can not be used with any xyz logical block
+device.
+
+The assumption/thought behind modifying IO scheduler is that resource control
+is needed only on leaf nodes where the actual contention for resources is
+present and not on intertermediate logical block devices.
+
+Consider following hypothetical scenario. Lets say there are three physical
+disks, namely sda, sdb and sdc. Two logical volumes (lv0 and lv1) have been
+created on top of these. Some part of sdb is in lv0 and some part is in lv1.
+
+			    lv0      lv1
+			  /	\  /     \
+			sda      sdb      sdc
+
+Also consider following cgroup hierarchy
+
+				root
+				/   \
+			       A     B
+			      / \    / \
+			     T1 T2  T3  T4
+
+A and B are two cgroups and T1, T2, T3 and T4 are tasks with-in those cgroups.
+Assuming T1, T2, T3 and T4 are doing IO on lv0 and lv1. These tasks should
+get their fair share of bandwidth on disks sda, sdb and sdc. There is no
+IO control on intermediate logical block nodes (lv0, lv1).
+
+So if tasks T1 and T2 are doing IO on lv0 and T3 and T4 are doing IO on lv1
+only, there will not be any contetion for resources between group A and B if
+IO is going to sda or sdc. But if actual IO gets translated to disk sdb, then
+IO scheduler associated with the sdb will distribute disk bandwidth to
+group A and B proportionate to their weight.
+
+CFQ already has the notion of fairness and it provides differential disk
+access based on priority and class of the task. Just that it is flat and
+with cgroup stuff, it needs to be made hierarchical.
+
+Rest of the IO schedulers (noop, deadline and AS) don't have any notion
+of fairness among various threads.
+
+One of the concerns raised with modifying IO schedulers was that we don't
+want to replicate the code in all the IO schedulers. These patches share
+the fair queuing code which has been moved to a common layer (elevator
+layer). Hence we don't end up replicating code across IO schedulers.
+
+Design
+======
+This patchset primarily uses BFQ (Budget Fair Queuing) code to provide
+fairness among different IO queues. Fabio and Paolo implemented BFQ which uses
+B-WF2Q+ algorithm for fair queuing.
+
+Why BFQ?
+
+- Not sure if weighted round robin logic of CFQ can be easily extended for
+  hierarchical mode. One of the things is that we can not keep dividing
+  the time slice of parent group among childrens. Deeper we go in hierarchy
+  time slice will get smaller.
+
+  One of the ways to implement hierarchical support could be to keep track
+  of virtual time and service provided to queue/group and select a queue/group
+  for service based on any of the various available algoriths.
+
+  BFQ already had support for hierarchical scheduling, taking those patches
+  was easier.
+
+- BFQ was designed to provide tighter bounds/delay w.r.t service provided
+  to a queue. Delay/Jitter with BFQ is supposed to be O(1).
+
+  Note: BFQ originally used amount of IO done (number of sectors) as notion
+        of service provided. IOW, it tried to provide fairness in terms of
+        actual IO done and not in terms of actual time disk access was
+	given to a queue.
+
+	This patcheset modified BFQ to provide fairness in time domain because
+	that's what CFQ does. So idea was try not to deviate too much from
+	the CFQ behavior initially.
+
+	Providing fairness in time domain makes accounting trciky because
+	due to command queueing, at one time there might be multiple requests
+	from different queues and there is no easy way to find out how much
+	disk time actually was consumed by the requests of a particular
+	queue. More about this in comments in source code.
+
+So it is yet to be seen if changing to time domain still retains BFQ gurantees
+or not.
+
+From data structure point of view, one can think of a tree per device, where
+io groups and io queues are hanging and are being scheduled using B-WF2Q+
+algorithm. io_queue, is end queue where requests are actually stored and
+dispatched from (like cfqq).
+
+These io queues are primarily created by and managed by end io schedulers
+depending on its semantics. For example, noop, deadline and AS ioschedulers
+keep one io queues per cgroup and cfqq keeps one io queue per io_context in
+a cgroup (apart from async queues).
+
+A request is mapped to an io group by elevator layer and which io queue it
+is mapped to with in group depends on ioscheduler. Currently "current" task
+is used to determine the cgroup (hence io group) of the request. Down the
+line we need to make use of bio-cgroup patches to map delayed writes to
+right group.
+
+Going back to old behavior
+==========================
+In new scheme of things essentially we are creating hierarchical fair
+queuing logic in elevator layer and chaning IO schedulers to make use of
+that logic so that end IO schedulers start supporting hierarchical scheduling.
+
+Elevator layer continues to support the old interfaces. So even if fair queuing
+is enabled at elevator layer, one can have both new hierchical scheduler as
+well as old non-hierarchical scheduler operating.
+
+Also noop, deadline and AS have option of enabling hierarchical scheduling.
+If it is selected, fair queuing is done in hierarchical manner. If hierarchical
+scheduling is disabled, noop, deadline and AS should retain their existing
+behavior.
+
+CFQ is the only exception where one can not disable fair queuing as it is
+needed for provding fairness among various threads even in non-hierarchical
+mode.
+
+Various user visible config options
+===================================
+CONFIG_IOSCHED_NOOP_HIER
+	- Enables hierchical fair queuing in noop. Not selecting this option
+	  leads to old behavior of noop.
+
+CONFIG_IOSCHED_DEADLINE_HIER
+	- Enables hierchical fair queuing in deadline. Not selecting this
+	  option leads to old behavior of deadline.
+
+CONFIG_IOSCHED_AS_HIER
+	- Enables hierchical fair queuing in AS. Not selecting this option
+	  leads to old behavior of AS.
+
+CONFIG_IOSCHED_CFQ_HIER
+	- Enables hierarchical fair queuing in CFQ. Not selecting this option
+	  still does fair queuing among various queus but it is flat and not
+	  hierarchical.
+
+Config options selected automatically
+=====================================
+These config options are not user visible and are selected/deselected
+automatically based on IO scheduler configurations.
+
+CONFIG_ELV_FAIR_QUEUING
+	- Enables/Disables the fair queuing logic at elevator layer.
+
+CONFIG_GROUP_IOSCHED
+	- Enables/Disables hierarchical queuing and associated cgroup bits.
+
+TODO
+====
+- Lots of cleanups, testing, bug fixing, optimizations, benchmarking etc...
+- Convert cgroup ioprio to notion of weight.
+- Anticipatory code will need more work. It is not working properly currently
+  and needs more thought.
+- Use of bio-cgroup patches.
+- Use of Nauman's per cgroup request descriptor patches.
+
+HOWTO
+=====
+So far I have done very simple testing of running two dd threads in two
+different cgroups. Here is what you can do.
+
+- Enable hierarchical scheduling in io scheuduler of your choice (say cfq).
+	CONFIG_IOSCHED_CFQ_HIER=y
+
+- Compile and boot into kernel and mount IO controller.
+
+	mount -t cgroup -o io none /cgroup
+
+- Create two cgroups
+	mkdir -p /cgroup/test1/ /cgroup/test2
+
+- Set io priority of group test1 and test2
+	echo 0 > /cgroup/test1/io.ioprio
+	echo 4 > /cgroup/test2/io.ioprio
+
+- Create two same size files (say 512MB each) on same disk (file1, file2) and
+  launch two dd threads in different cgroup to read those files. Make sure
+  right io scheduler is being used for the block device where files are
+  present (the one you compiled in hierarchical mode).
+
+	echo 1 > /proc/sys/vm/drop_caches
+
+	dd if=/mnt/lv0/zerofile1 of=/dev/null &
+	echo $! > /cgroup/test1/tasks
+	cat /cgroup/test1/tasks
+
+	dd if=/mnt/lv0/zerofile2 of=/dev/null &
+	echo $! > /cgroup/test2/tasks
+	cat /cgroup/test2/tasks
+
+- First dd should finish first.
+
+Some Test Results
+=================

Documentation/block/io-controller.txt line 221

Quoted 10 lines
Documentation/block/io-controller.txt new file +10 -0
+- Two dd in two cgroups with prio 0 and 4. Ran two "dd" in those cgroups.
+
+234179072 bytes (234 MB) copied, 10.1811 s, 23.0 MB/s
+234179072 bytes (234 MB) copied, 12.6187 s, 18.6 MB/s
+
+- Three dd in three cgroups with prio 0, 4, 4.
+
+234179072 bytes (234 MB) copied, 13.7654 s, 17.0 MB/s
+234179072 bytes (234 MB) copied, 19.476 s, 12.0 MB/s
+234179072 bytes (234 MB) copied, 20.1858 s, 11.6 MB/s

Hi Vivek,

I would be interested in knowing if these are the results expected?

Signature · 2 lines
regards,
Dhaval

A message that is not in the archive (sent off-list or not yet received): 20090312100054.GA8024-23VcF4HTsmIX0ybBhKVfKdBPR1lH4CV8@public.gmane.org

#64 lore raw

Re: [PATCH 01/10] Documentation

Quoted 239 lines · Dhaval Giani
Quoted 225 lines · Vivek Goyal

o Documentation for io-controller.

---
 Documentation/block/io-controller.txt |  221 +++++++++++++++++++++++++++++++++
 1 files changed, 221 insertions(+), 0 deletions(-)
 create mode 100644 Documentation/block/io-controller.txt
Documentation/block/io-controller.txt new file +211 -0
new file mode 100644
index 0000000..8884c5a
@@ -0,0 +1,221 @@
+				IO Controller
+				=============
+
+Overview
+========
+
+This patchset implements a proportional weight IO controller. That is one
+can create cgroups and assign prio/weights to those cgroups and task group
+will get access to disk proportionate to the weight of the group.
+
+These patches modify elevator layer and individual IO schedulers to do
+IO control hence this io controller works only on block devices which use
+one of the standard io schedulers can not be used with any xyz logical block
+device.
+
+The assumption/thought behind modifying IO scheduler is that resource control
+is needed only on leaf nodes where the actual contention for resources is
+present and not on intertermediate logical block devices.
+
+Consider following hypothetical scenario. Lets say there are three physical
+disks, namely sda, sdb and sdc. Two logical volumes (lv0 and lv1) have been
+created on top of these. Some part of sdb is in lv0 and some part is in lv1.
+
+			    lv0      lv1
+			  /	\  /     \
+			sda      sdb      sdc
+
+Also consider following cgroup hierarchy
+
+				root
+				/   \
+			       A     B
+			      / \    / \
+			     T1 T2  T3  T4
+
+A and B are two cgroups and T1, T2, T3 and T4 are tasks with-in those cgroups.
+Assuming T1, T2, T3 and T4 are doing IO on lv0 and lv1. These tasks should
+get their fair share of bandwidth on disks sda, sdb and sdc. There is no
+IO control on intermediate logical block nodes (lv0, lv1).
+
+So if tasks T1 and T2 are doing IO on lv0 and T3 and T4 are doing IO on lv1
+only, there will not be any contetion for resources between group A and B if
+IO is going to sda or sdc. But if actual IO gets translated to disk sdb, then
+IO scheduler associated with the sdb will distribute disk bandwidth to
+group A and B proportionate to their weight.
+
+CFQ already has the notion of fairness and it provides differential disk
+access based on priority and class of the task. Just that it is flat and
+with cgroup stuff, it needs to be made hierarchical.
+
+Rest of the IO schedulers (noop, deadline and AS) don't have any notion
+of fairness among various threads.
+
+One of the concerns raised with modifying IO schedulers was that we don't
+want to replicate the code in all the IO schedulers. These patches share
+the fair queuing code which has been moved to a common layer (elevator
+layer). Hence we don't end up replicating code across IO schedulers.
+
+Design
+======
+This patchset primarily uses BFQ (Budget Fair Queuing) code to provide
+fairness among different IO queues. Fabio and Paolo implemented BFQ which uses
+B-WF2Q+ algorithm for fair queuing.
+
+Why BFQ?
+
+- Not sure if weighted round robin logic of CFQ can be easily extended for
+  hierarchical mode. One of the things is that we can not keep dividing
+  the time slice of parent group among childrens. Deeper we go in hierarchy
+  time slice will get smaller.
+
+  One of the ways to implement hierarchical support could be to keep track
+  of virtual time and service provided to queue/group and select a queue/group
+  for service based on any of the various available algoriths.
+
+  BFQ already had support for hierarchical scheduling, taking those patches
+  was easier.
+
+- BFQ was designed to provide tighter bounds/delay w.r.t service provided
+  to a queue. Delay/Jitter with BFQ is supposed to be O(1).
+
+  Note: BFQ originally used amount of IO done (number of sectors) as notion
+        of service provided. IOW, it tried to provide fairness in terms of
+        actual IO done and not in terms of actual time disk access was
+	given to a queue.
+
+	This patcheset modified BFQ to provide fairness in time domain because
+	that's what CFQ does. So idea was try not to deviate too much from
+	the CFQ behavior initially.
+
+	Providing fairness in time domain makes accounting trciky because
+	due to command queueing, at one time there might be multiple requests
+	from different queues and there is no easy way to find out how much
+	disk time actually was consumed by the requests of a particular
+	queue. More about this in comments in source code.
+
+So it is yet to be seen if changing to time domain still retains BFQ gurantees
+or not.
+
+From data structure point of view, one can think of a tree per device, where
+io groups and io queues are hanging and are being scheduled using B-WF2Q+
+algorithm. io_queue, is end queue where requests are actually stored and
+dispatched from (like cfqq).
+
+These io queues are primarily created by and managed by end io schedulers
+depending on its semantics. For example, noop, deadline and AS ioschedulers
+keep one io queues per cgroup and cfqq keeps one io queue per io_context in
+a cgroup (apart from async queues).
+
+A request is mapped to an io group by elevator layer and which io queue it
+is mapped to with in group depends on ioscheduler. Currently "current" task
+is used to determine the cgroup (hence io group) of the request. Down the
+line we need to make use of bio-cgroup patches to map delayed writes to
+right group.
+
+Going back to old behavior
+==========================
+In new scheme of things essentially we are creating hierarchical fair
+queuing logic in elevator layer and chaning IO schedulers to make use of
+that logic so that end IO schedulers start supporting hierarchical scheduling.
+
+Elevator layer continues to support the old interfaces. So even if fair queuing
+is enabled at elevator layer, one can have both new hierchical scheduler as
+well as old non-hierarchical scheduler operating.
+
+Also noop, deadline and AS have option of enabling hierarchical scheduling.
+If it is selected, fair queuing is done in hierarchical manner. If hierarchical
+scheduling is disabled, noop, deadline and AS should retain their existing
+behavior.
+
+CFQ is the only exception where one can not disable fair queuing as it is
+needed for provding fairness among various threads even in non-hierarchical
+mode.
+
+Various user visible config options
+===================================
+CONFIG_IOSCHED_NOOP_HIER
+	- Enables hierchical fair queuing in noop. Not selecting this option
+	  leads to old behavior of noop.
+
+CONFIG_IOSCHED_DEADLINE_HIER
+	- Enables hierchical fair queuing in deadline. Not selecting this
+	  option leads to old behavior of deadline.
+
+CONFIG_IOSCHED_AS_HIER
+	- Enables hierchical fair queuing in AS. Not selecting this option
+	  leads to old behavior of AS.
+
+CONFIG_IOSCHED_CFQ_HIER
+	- Enables hierarchical fair queuing in CFQ. Not selecting this option
+	  still does fair queuing among various queus but it is flat and not
+	  hierarchical.
+
+Config options selected automatically
+=====================================
+These config options are not user visible and are selected/deselected
+automatically based on IO scheduler configurations.
+
+CONFIG_ELV_FAIR_QUEUING
+	- Enables/Disables the fair queuing logic at elevator layer.
+
+CONFIG_GROUP_IOSCHED
+	- Enables/Disables hierarchical queuing and associated cgroup bits.
+
+TODO
+====
+- Lots of cleanups, testing, bug fixing, optimizations, benchmarking etc...
+- Convert cgroup ioprio to notion of weight.
+- Anticipatory code will need more work. It is not working properly currently
+  and needs more thought.
+- Use of bio-cgroup patches.
+- Use of Nauman's per cgroup request descriptor patches.
+
+HOWTO
+=====
+So far I have done very simple testing of running two dd threads in two
+different cgroups. Here is what you can do.
+
+- Enable hierarchical scheduling in io scheuduler of your choice (say cfq).
+	CONFIG_IOSCHED_CFQ_HIER=y
+
+- Compile and boot into kernel and mount IO controller.
+
+	mount -t cgroup -o io none /cgroup
+
+- Create two cgroups
+	mkdir -p /cgroup/test1/ /cgroup/test2
+
+- Set io priority of group test1 and test2
+	echo 0 > /cgroup/test1/io.ioprio
+	echo 4 > /cgroup/test2/io.ioprio
+
+- Create two same size files (say 512MB each) on same disk (file1, file2) and
+  launch two dd threads in different cgroup to read those files. Make sure
+  right io scheduler is being used for the block device where files are
+  present (the one you compiled in hierarchical mode).
+
+	echo 1 > /proc/sys/vm/drop_caches
+
+	dd if=/mnt/lv0/zerofile1 of=/dev/null &
+	echo $! > /cgroup/test1/tasks
+	cat /cgroup/test1/tasks
+
+	dd if=/mnt/lv0/zerofile2 of=/dev/null &
+	echo $! > /cgroup/test2/tasks
+	cat /cgroup/test2/tasks
+
+- First dd should finish first.
+
+Some Test Results
+=================

Documentation/block/io-controller.txt line 221

Quoted 10 lines
Documentation/block/io-controller.txt new file +10 -0
+- Two dd in two cgroups with prio 0 and 4. Ran two "dd" in those cgroups.
+
+234179072 bytes (234 MB) copied, 10.1811 s, 23.0 MB/s
+234179072 bytes (234 MB) copied, 12.6187 s, 18.6 MB/s
+
+- Three dd in three cgroups with prio 0, 4, 4.
+
+234179072 bytes (234 MB) copied, 13.7654 s, 17.0 MB/s
+234179072 bytes (234 MB) copied, 19.476 s, 12.0 MB/s
+234179072 bytes (234 MB) copied, 20.1858 s, 11.6 MB/s

Hi Vivek,

Quoted 2 lines

I would be interested in knowing if these are the results expected?

Hi Dhaval,

Good question. Keeping current expectation in mind, yes these are expected results. To begin with, current expectations are that try to emulate cfq behavior and the kind of service differentiation we get between threads of different priority, same kind of service differentiation we should get from different cgroups.

Having said that, in theory a more accurate estimate should be amount of actual disk time a queue/cgroup got. I have put a tracing message to keep track of total service received by a queue. If you run "blktrace" then you can see that. Ideally, total service received by two threads over a period of time should be in same proportion as their cgroup weights.

It will not be easy to achive it given the constraints we have got in terms of how to accurately we can account for disk time actually used by a queue in certain situations. So to begin with I am targetting that try to meet same kind of service differentation between cgroups as cfq provides between threads and then slowly refine it to see how close one can come to get accurate numbers in terms of "total_serivce" received by each queue.

Thanks
Vivek

#65 lore raw

Re: [PATCH 01/10] Documentation

In reply to Dhaval Giani
Quoted 239 lines · Dhaval Giani
Quoted 225 lines · Vivek Goyal

o Documentation for io-controller.

---
 Documentation/block/io-controller.txt |  221 +++++++++++++++++++++++++++++++++
 1 files changed, 221 insertions(+), 0 deletions(-)
 create mode 100644 Documentation/block/io-controller.txt
Documentation/block/io-controller.txt new file +211 -0
new file mode 100644
index 0000000..8884c5a
@@ -0,0 +1,221 @@
+				IO Controller
+				=============
+
+Overview
+========
+
+This patchset implements a proportional weight IO controller. That is one
+can create cgroups and assign prio/weights to those cgroups and task group
+will get access to disk proportionate to the weight of the group.
+
+These patches modify elevator layer and individual IO schedulers to do
+IO control hence this io controller works only on block devices which use
+one of the standard io schedulers can not be used with any xyz logical block
+device.
+
+The assumption/thought behind modifying IO scheduler is that resource control
+is needed only on leaf nodes where the actual contention for resources is
+present and not on intertermediate logical block devices.
+
+Consider following hypothetical scenario. Lets say there are three physical
+disks, namely sda, sdb and sdc. Two logical volumes (lv0 and lv1) have been
+created on top of these. Some part of sdb is in lv0 and some part is in lv1.
+
+			    lv0      lv1
+			  /	\  /     \
+			sda      sdb      sdc
+
+Also consider following cgroup hierarchy
+
+				root
+				/   \
+			       A     B
+			      / \    / \
+			     T1 T2  T3  T4
+
+A and B are two cgroups and T1, T2, T3 and T4 are tasks with-in those cgroups.
+Assuming T1, T2, T3 and T4 are doing IO on lv0 and lv1. These tasks should
+get their fair share of bandwidth on disks sda, sdb and sdc. There is no
+IO control on intermediate logical block nodes (lv0, lv1).
+
+So if tasks T1 and T2 are doing IO on lv0 and T3 and T4 are doing IO on lv1
+only, there will not be any contetion for resources between group A and B if
+IO is going to sda or sdc. But if actual IO gets translated to disk sdb, then
+IO scheduler associated with the sdb will distribute disk bandwidth to
+group A and B proportionate to their weight.
+
+CFQ already has the notion of fairness and it provides differential disk
+access based on priority and class of the task. Just that it is flat and
+with cgroup stuff, it needs to be made hierarchical.
+
+Rest of the IO schedulers (noop, deadline and AS) don't have any notion
+of fairness among various threads.
+
+One of the concerns raised with modifying IO schedulers was that we don't
+want to replicate the code in all the IO schedulers. These patches share
+the fair queuing code which has been moved to a common layer (elevator
+layer). Hence we don't end up replicating code across IO schedulers.
+
+Design
+======
+This patchset primarily uses BFQ (Budget Fair Queuing) code to provide
+fairness among different IO queues. Fabio and Paolo implemented BFQ which uses
+B-WF2Q+ algorithm for fair queuing.
+
+Why BFQ?
+
+- Not sure if weighted round robin logic of CFQ can be easily extended for
+  hierarchical mode. One of the things is that we can not keep dividing
+  the time slice of parent group among childrens. Deeper we go in hierarchy
+  time slice will get smaller.
+
+  One of the ways to implement hierarchical support could be to keep track
+  of virtual time and service provided to queue/group and select a queue/group
+  for service based on any of the various available algoriths.
+
+  BFQ already had support for hierarchical scheduling, taking those patches
+  was easier.
+
+- BFQ was designed to provide tighter bounds/delay w.r.t service provided
+  to a queue. Delay/Jitter with BFQ is supposed to be O(1).
+
+  Note: BFQ originally used amount of IO done (number of sectors) as notion
+        of service provided. IOW, it tried to provide fairness in terms of
+        actual IO done and not in terms of actual time disk access was
+	given to a queue.
+
+	This patcheset modified BFQ to provide fairness in time domain because
+	that's what CFQ does. So idea was try not to deviate too much from
+	the CFQ behavior initially.
+
+	Providing fairness in time domain makes accounting trciky because
+	due to command queueing, at one time there might be multiple requests
+	from different queues and there is no easy way to find out how much
+	disk time actually was consumed by the requests of a particular
+	queue. More about this in comments in source code.
+
+So it is yet to be seen if changing to time domain still retains BFQ gurantees
+or not.
+
+From data structure point of view, one can think of a tree per device, where
+io groups and io queues are hanging and are being scheduled using B-WF2Q+
+algorithm. io_queue, is end queue where requests are actually stored and
+dispatched from (like cfqq).
+
+These io queues are primarily created by and managed by end io schedulers
+depending on its semantics. For example, noop, deadline and AS ioschedulers
+keep one io queues per cgroup and cfqq keeps one io queue per io_context in
+a cgroup (apart from async queues).
+
+A request is mapped to an io group by elevator layer and which io queue it
+is mapped to with in group depends on ioscheduler. Currently "current" task
+is used to determine the cgroup (hence io group) of the request. Down the
+line we need to make use of bio-cgroup patches to map delayed writes to
+right group.
+
+Going back to old behavior
+==========================
+In new scheme of things essentially we are creating hierarchical fair
+queuing logic in elevator layer and chaning IO schedulers to make use of
+that logic so that end IO schedulers start supporting hierarchical scheduling.
+
+Elevator layer continues to support the old interfaces. So even if fair queuing
+is enabled at elevator layer, one can have both new hierchical scheduler as
+well as old non-hierarchical scheduler operating.
+
+Also noop, deadline and AS have option of enabling hierarchical scheduling.
+If it is selected, fair queuing is done in hierarchical manner. If hierarchical
+scheduling is disabled, noop, deadline and AS should retain their existing
+behavior.
+
+CFQ is the only exception where one can not disable fair queuing as it is
+needed for provding fairness among various threads even in non-hierarchical
+mode.
+
+Various user visible config options
+===================================
+CONFIG_IOSCHED_NOOP_HIER
+	- Enables hierchical fair queuing in noop. Not selecting this option
+	  leads to old behavior of noop.
+
+CONFIG_IOSCHED_DEADLINE_HIER
+	- Enables hierchical fair queuing in deadline. Not selecting this
+	  option leads to old behavior of deadline.
+
+CONFIG_IOSCHED_AS_HIER
+	- Enables hierchical fair queuing in AS. Not selecting this option
+	  leads to old behavior of AS.
+
+CONFIG_IOSCHED_CFQ_HIER
+	- Enables hierarchical fair queuing in CFQ. Not selecting this option
+	  still does fair queuing among various queus but it is flat and not
+	  hierarchical.
+
+Config options selected automatically
+=====================================
+These config options are not user visible and are selected/deselected
+automatically based on IO scheduler configurations.
+
+CONFIG_ELV_FAIR_QUEUING
+	- Enables/Disables the fair queuing logic at elevator layer.
+
+CONFIG_GROUP_IOSCHED
+	- Enables/Disables hierarchical queuing and associated cgroup bits.
+
+TODO
+====
+- Lots of cleanups, testing, bug fixing, optimizations, benchmarking etc...
+- Convert cgroup ioprio to notion of weight.
+- Anticipatory code will need more work. It is not working properly currently
+  and needs more thought.
+- Use of bio-cgroup patches.
+- Use of Nauman's per cgroup request descriptor patches.
+
+HOWTO
+=====
+So far I have done very simple testing of running two dd threads in two
+different cgroups. Here is what you can do.
+
+- Enable hierarchical scheduling in io scheuduler of your choice (say cfq).
+	CONFIG_IOSCHED_CFQ_HIER=y
+
+- Compile and boot into kernel and mount IO controller.
+
+	mount -t cgroup -o io none /cgroup
+
+- Create two cgroups
+	mkdir -p /cgroup/test1/ /cgroup/test2
+
+- Set io priority of group test1 and test2
+	echo 0 > /cgroup/test1/io.ioprio
+	echo 4 > /cgroup/test2/io.ioprio
+
+- Create two same size files (say 512MB each) on same disk (file1, file2) and
+  launch two dd threads in different cgroup to read those files. Make sure
+  right io scheduler is being used for the block device where files are
+  present (the one you compiled in hierarchical mode).
+
+	echo 1 > /proc/sys/vm/drop_caches
+
+	dd if=/mnt/lv0/zerofile1 of=/dev/null &
+	echo $! > /cgroup/test1/tasks
+	cat /cgroup/test1/tasks
+
+	dd if=/mnt/lv0/zerofile2 of=/dev/null &
+	echo $! > /cgroup/test2/tasks
+	cat /cgroup/test2/tasks
+
+- First dd should finish first.
+
+Some Test Results
+=================

Documentation/block/io-controller.txt line 221

Quoted 10 lines
Documentation/block/io-controller.txt new file +10 -0
+- Two dd in two cgroups with prio 0 and 4. Ran two "dd" in those cgroups.
+
+234179072 bytes (234 MB) copied, 10.1811 s, 23.0 MB/s
+234179072 bytes (234 MB) copied, 12.6187 s, 18.6 MB/s
+
+- Three dd in three cgroups with prio 0, 4, 4.
+
+234179072 bytes (234 MB) copied, 13.7654 s, 17.0 MB/s
+234179072 bytes (234 MB) copied, 19.476 s, 12.0 MB/s
+234179072 bytes (234 MB) copied, 20.1858 s, 11.6 MB/s

Hi Vivek,

Quoted 2 lines

I would be interested in knowing if these are the results expected?

Hi Dhaval,

Good question. Keeping current expectation in mind, yes these are expected results. To begin with, current expectations are that try to emulate cfq behavior and the kind of service differentiation we get between threads of different priority, same kind of service differentiation we should get from different cgroups.

Having said that, in theory a more accurate estimate should be amount of actual disk time a queue/cgroup got. I have put a tracing message to keep track of total service received by a queue. If you run "blktrace" then you can see that. Ideally, total service received by two threads over a period of time should be in same proportion as their cgroup weights.

It will not be easy to achive it given the constraints we have got in terms of how to accurately we can account for disk time actually used by a queue in certain situations. So to begin with I am targetting that try to meet same kind of service differentation between cgroups as cfq provides between threads and then slowly refine it to see how close one can come to get accurate numbers in terms of "total_serivce" received by each queue.

Thanks
Vivek

#66 lore raw

Re: [PATCH 01/10] Documentation

Quoted 4 lines

From: Vivek Goyal <[email protected]>
Date: Thu, Mar 12, 2009 10:04:50AM -0400

On Thu, Mar 12, 2009 at 03:30:54PM +0530, Dhaval Giani wrote:

...

Quoted 33 lines
Quoted 15 lines

Documentation/block/io-controller.txt line 221

Quoted 12 lines
Documentation/block/io-controller.txt new file +12 -0
+Some Test Results
+=================
+- Two dd in two cgroups with prio 0 and 4. Ran two "dd" in those cgroups.
+
+234179072 bytes (234 MB) copied, 10.1811 s, 23.0 MB/s
+234179072 bytes (234 MB) copied, 12.6187 s, 18.6 MB/s
+
+- Three dd in three cgroups with prio 0, 4, 4.
+
+234179072 bytes (234 MB) copied, 13.7654 s, 17.0 MB/s
+234179072 bytes (234 MB) copied, 19.476 s, 12.0 MB/s
+234179072 bytes (234 MB) copied, 20.1858 s, 11.6 MB/s

Hi Vivek,

Quoted 2 lines

I would be interested in knowing if these are the results expected?

Hi Dhaval,

Good question. Keeping current expectation in mind, yes these are expected results. To begin with, current expectations are that try to emulate cfq behavior and the kind of service differentiation we get between threads of different priority, same kind of service differentiation we should get from different cgroups.

Having said that, in theory a more accurate estimate should be amount of actual disk time a queue/cgroup got. I have put a tracing message to keep track of total service received by a queue. If you run "blktrace" then you can see that. Ideally, total service received by two threads over a period of time should be in same proportion as their cgroup weights.

Quoted 8 lines

It will not be easy to achive it given the constraints we have got in terms of how to accurately we can account for disk time actually used by a queue in certain situations. So to begin with I am targetting that try to meet same kind of service differentation between cgroups as cfq provides between threads and then slowly refine it to see how close one can come to get accurate numbers in terms of "total_serivce" received by each queue.

There is also another issue to consider; to achieve a proper weighted distribution of ``service time'' (assuming that service time can be attributed accurately) over any time window, we need also that the tasks actually compete for disk service during this window.

For example, in the case above with three tasks, the highest weight task terminates earlier than the other ones, so we have two time frames:
during the first one disk time is divided among all the three tasks according to their weights, then the highest weight one terminates, and disk time is divided (equally) among the remaining ones.

A message that is not in the archive (sent off-list or not yet received): [email protected]

#68 lore raw

Re: [PATCH 01/10] Documentation

Quoted 52 lines · Fabio Checconi
Quoted 4 lines

From: Vivek Goyal <[email protected]>
Date: Thu, Mar 12, 2009 10:04:50AM -0400

On Thu, Mar 12, 2009 at 03:30:54PM +0530, Dhaval Giani wrote:

...

Quoted 33 lines
Quoted 15 lines

Documentation/block/io-controller.txt line 221

Quoted 12 lines
Documentation/block/io-controller.txt new file +12 -0
+Some Test Results
+=================
+- Two dd in two cgroups with prio 0 and 4. Ran two "dd" in those cgroups.
+
+234179072 bytes (234 MB) copied, 10.1811 s, 23.0 MB/s
+234179072 bytes (234 MB) copied, 12.6187 s, 18.6 MB/s
+
+- Three dd in three cgroups with prio 0, 4, 4.
+
+234179072 bytes (234 MB) copied, 13.7654 s, 17.0 MB/s
+234179072 bytes (234 MB) copied, 19.476 s, 12.0 MB/s
+234179072 bytes (234 MB) copied, 20.1858 s, 11.6 MB/s

Hi Vivek,

Quoted 2 lines

I would be interested in knowing if these are the results expected?

Hi Dhaval,

Good question. Keeping current expectation in mind, yes these are expected results. To begin with, current expectations are that try to emulate cfq behavior and the kind of service differentiation we get between threads of different priority, same kind of service differentiation we should get from different cgroups.

Having said that, in theory a more accurate estimate should be amount of actual disk time a queue/cgroup got. I have put a tracing message to keep track of total service received by a queue. If you run "blktrace" then you can see that. Ideally, total service received by two threads over a period of time should be in same proportion as their cgroup weights.

Quoted 8 lines

It will not be easy to achive it given the constraints we have got in terms of how to accurately we can account for disk time actually used by a queue in certain situations. So to begin with I am targetting that try to meet same kind of service differentation between cgroups as cfq provides between threads and then slowly refine it to see how close one can come to get accurate numbers in terms of "total_serivce" received by each queue.

There is also another issue to consider; to achieve a proper weighted distribution of ``service time'' (assuming that service time can be attributed accurately) over any time window, we need also that the tasks actually compete for disk service during this window.

Quoted 5 lines

For example, in the case above with three tasks, the highest weight task terminates earlier than the other ones, so we have two time frames:
during the first one disk time is divided among all the three tasks according to their weights, then the highest weight one terminates, and disk time is divided (equally) among the remaining ones.

True. But we can do one thing. I am printing total_service every time a queue expires(elv_ioq_served()). So when first task exits, at that point of time, we can see how much service each competing queue has received till that point and it should be proportionate to queue's weight.

Thanks
Vivek

#69 lore raw

Re: [PATCH 01/10] Documentation

In reply to Fabio Checconi
Quoted 52 lines · Fabio Checconi
Quoted 4 lines

From: Vivek Goyal <[email protected]>
Date: Thu, Mar 12, 2009 10:04:50AM -0400

On Thu, Mar 12, 2009 at 03:30:54PM +0530, Dhaval Giani wrote:

...

Quoted 33 lines
Quoted 15 lines

Documentation/block/io-controller.txt line 221

Quoted 12 lines
Documentation/block/io-controller.txt new file +12 -0
+Some Test Results
+=================
+- Two dd in two cgroups with prio 0 and 4. Ran two "dd" in those cgroups.
+
+234179072 bytes (234 MB) copied, 10.1811 s, 23.0 MB/s
+234179072 bytes (234 MB) copied, 12.6187 s, 18.6 MB/s
+
+- Three dd in three cgroups with prio 0, 4, 4.
+
+234179072 bytes (234 MB) copied, 13.7654 s, 17.0 MB/s
+234179072 bytes (234 MB) copied, 19.476 s, 12.0 MB/s
+234179072 bytes (234 MB) copied, 20.1858 s, 11.6 MB/s

Hi Vivek,

Quoted 2 lines

I would be interested in knowing if these are the results expected?

Hi Dhaval,

Good question. Keeping current expectation in mind, yes these are expected results. To begin with, current expectations are that try to emulate cfq behavior and the kind of service differentiation we get between threads of different priority, same kind of service differentiation we should get from different cgroups.

Having said that, in theory a more accurate estimate should be amount of actual disk time a queue/cgroup got. I have put a tracing message to keep track of total service received by a queue. If you run "blktrace" then you can see that. Ideally, total service received by two threads over a period of time should be in same proportion as their cgroup weights.

Quoted 8 lines

It will not be easy to achive it given the constraints we have got in terms of how to accurately we can account for disk time actually used by a queue in certain situations. So to begin with I am targetting that try to meet same kind of service differentation between cgroups as cfq provides between threads and then slowly refine it to see how close one can come to get accurate numbers in terms of "total_serivce" received by each queue.

There is also another issue to consider; to achieve a proper weighted distribution of ``service time'' (assuming that service time can be attributed accurately) over any time window, we need also that the tasks actually compete for disk service during this window.

Quoted 5 lines

For example, in the case above with three tasks, the highest weight task terminates earlier than the other ones, so we have two time frames:
during the first one disk time is divided among all the three tasks according to their weights, then the highest weight one terminates, and disk time is divided (equally) among the remaining ones.

True. But we can do one thing. I am printing total_service every time a queue expires(elv_ioq_served()). So when first task exits, at that point of time, we can see how much service each competing queue has received till that point and it should be proportionate to queue's weight.

Thanks
Vivek

A message that is not in the archive (sent off-list or not yet received): [email protected]

#71 lore raw

Re: [PATCH 01/10] Documentation

Quoted 20 lines · Vivek Goyal
Quoted 4 lines

Hi Vivek,

I would be interested in knowing if these are the results expected?

Hi Dhaval,

Good question. Keeping current expectation in mind, yes these are expected results. To begin with, current expectations are that try to emulate cfq behavior and the kind of service differentiation we get between threads of different priority, same kind of service differentiation we should get from different cgroups.

Having said that, in theory a more accurate estimate should be amount of actual disk time a queue/cgroup got. I have put a tracing message to keep track of total service received by a queue. If you run "blktrace" then you can see that. Ideally, total service received by two threads over a period of time should be in same proportion as their cgroup weights.

Quoted 7 lines

It will not be easy to achive it given the constraints we have got in terms of how to accurately we can account for disk time actually used by a queue in certain situations. So to begin with I am targetting that try to meet same kind of service differentation between cgroups as cfq provides between threads and then slowly refine it to see how close one can come to get accurate numbers in terms of "total_serivce" received by each queue.

Hi Vivek,

I simply tested with blktrace opened. I create two groups and set ioprio 4 and 7 respectively(the corresponding weight should 4:1, right?), and start two dd concurrently. UUIC, Ideally, the proportion of service two dd got should be 4:1 in a period of time when they are running. I extract *served* value from blktrace output and sum them up. I found the proportion of the sum of *served* value is about 1.7:1
Am i missing something?

I extract the following highlight value
8,0 0 0 18.914906549 0 m N 6601ioq served=*0x13* total service=0x184d

Quoted 6 lines

Thanks
Vivek

Signature · 2 lines
Regards
Gui Jianfeng
#72 lore raw

Re: [PATCH 01/10] Documentation

In reply to Vivek Goyal
Quoted 20 lines · Vivek Goyal
Quoted 4 lines

Hi Vivek,

I would be interested in knowing if these are the results expected?

Hi Dhaval,

Good question. Keeping current expectation in mind, yes these are expected results. To begin with, current expectations are that try to emulate cfq behavior and the kind of service differentiation we get between threads of different priority, same kind of service differentiation we should get from different cgroups.

Having said that, in theory a more accurate estimate should be amount of actual disk time a queue/cgroup got. I have put a tracing message to keep track of total service received by a queue. If you run "blktrace" then you can see that. Ideally, total service received by two threads over a period of time should be in same proportion as their cgroup weights.

Quoted 7 lines

It will not be easy to achive it given the constraints we have got in terms of how to accurately we can account for disk time actually used by a queue in certain situations. So to begin with I am targetting that try to meet same kind of service differentation between cgroups as cfq provides between threads and then slowly refine it to see how close one can come to get accurate numbers in terms of "total_serivce" received by each queue.

Hi Vivek,

I simply tested with blktrace opened. I create two groups and set ioprio 4 and 7 respectively(the corresponding weight should 4:1, right?), and start two dd concurrently. UUIC, Ideally, the proportion of service two dd got should be 4:1 in a period of time when they are running. I extract *served* value from blktrace output and sum them up. I found the proportion of the sum of *served* value is about 1.7:1
Am i missing something?

I extract the following highlight value
8,0 0 0 18.914906549 0 m N 6601ioq served=*0x13* total service=0x184d

Quoted 6 lines

Thanks
Vivek

Signature · 2 lines
Regards
Gui Jianfeng
#73 lore raw

Re: [PATCH 01/10] Documentation

Quoted 29 lines · Gui Jianfeng
Quoted 20 lines · Vivek Goyal
Quoted 4 lines

Hi Vivek,

I would be interested in knowing if these are the results expected?

Hi Dhaval,

Good question. Keeping current expectation in mind, yes these are expected results. To begin with, current expectations are that try to emulate cfq behavior and the kind of service differentiation we get between threads of different priority, same kind of service differentiation we should get from different cgroups.

Having said that, in theory a more accurate estimate should be amount of actual disk time a queue/cgroup got. I have put a tracing message to keep track of total service received by a queue. If you run "blktrace" then you can see that. Ideally, total service received by two threads over a period of time should be in same proportion as their cgroup weights.

Quoted 7 lines

It will not be easy to achive it given the constraints we have got in terms of how to accurately we can account for disk time actually used by a queue in certain situations. So to begin with I am targetting that try to meet same kind of service differentation between cgroups as cfq provides between threads and then slowly refine it to see how close one can come to get accurate numbers in terms of "total_serivce" received by each queue.

Quoted 4 lines

Hi Vivek,

I simply tested with blktrace opened. I create two groups and set ioprio 4 and 7 respectively(the corresponding weight should 4:1, right?),

Hi Gui,

Thanks for testing. You are right about weight proportions.

Quoted 6 lines

and
start two dd concurrently. UUIC, Ideally, the proportion of service two dd got should be 4:1 in a period of time when they are running. I extract *served* value from blktrace output and sum them up. I found the proportion of the sum of *served* value is about 1.7:1
Am i missing something?

Actually getting the service proportion in same ratio as weight proportion is quite hard for sync queues. The biggest issue is that many a times sync queues are not continuously backlogged and they do some IO and then dispatch a next round of requests.

Most of the time idling seems to be the solution for giving an impression that sync queue is continuously backlogged but it also has potential to reduce throughput on faster hardware.

Anyway, can you please send me your complete blkparse output. There are many a places where code has been designed to favor throughput than fairness. Looking at your blkparse output, will give me better idea what's the issue in your setup.

Also please try the attached patch. I have experimented with waiting for new request to come before sync queue is expired. It helps me in getting the fairness numbers at least with noop on non-queueing rotational media.

I also have introduced a new tunable "fairness". Above code will kick in only if this variable is set to 1. Many a places where we favor throughput over fairness, I plan to use this variable as condition to let user decide whether to choose fairness over throughput. I am not sure at how many places it really makes sense, but it atleast gives us something to play and compare the throughput in two cases.

This patch applies on my current tree after removing tomost patceh "anticipatory scheduling changes". My code has changed a bit since the posting, so you might have to message this patch a bit.

Thanks
Vivek

DESC
io-controller: idle for sometime on sync queue before expiring it EDESC

o When a sync queue expires, in many cases it might be empty and then it will be deleted from the active tree. This will lead to a scenario where out of two competing queues, only one is on the tree and when a new queue is selected, vtime jump takes place and we don't see services provided in proportion to weight.

o In general this is a fundamental problem with fairness of sync queues where queues are not continuously backlogged. Looks like idling is only solution to make sure such kind of queues can get some decent amount of disk bandwidth in the face of competion from continusouly backlogged queues. But excessive idling has potential to reduce performance on SSD and disks with commnad queuing.

o This patch experiments with waiting for next request to come before a queue is expired after it has consumed its time slice. This can ensure more accurate fairness numbers in some cases.

o Introduced a tunable "fairness". If set, io-controller will put more focus on getting fairness right than getting throughput right.

---
 block/blk-sysfs.c   |    7 ++++
 block/elevator-fq.c |   85 +++++++++++++++++++++++++++++++++++++++++++++-------
 block/elevator-fq.h |   12 +++++++
 3 files changed, 94 insertions(+), 10 deletions(-)
block/elevator-fq.h +12 -0
Index: linux1/block/elevator-fq.h
===================================================================
@@ -318,6 +318,13 @@ struct elv_fq_data {
 	unsigned long long rate_sampling_start; /*sampling window start jifies*/
 	/* number of sectors finished io during current sampling window */
 	unsigned long rate_sectors_current;
+
+	/*
+	 * If set to 1, will disable many optimizations done for boost
+	 * throughput and focus more on providing fairness for sync
+	 * queues.
+	 */
+	int fairness;
 };
 
 extern int elv_slice_idle;
@@ -340,6 +347,7 @@ enum elv_queue_state_flags {
 	ELV_QUEUE_FLAG_idle_window,	  /* elevator slice idling enabled */
 	ELV_QUEUE_FLAG_wait_request,	  /* waiting for a request */
 	ELV_QUEUE_FLAG_slice_new,	  /* no requests dispatched in slice */
+	ELV_QUEUE_FLAG_wait_busy,	  /* wait for this queue to get busy */
 	ELV_QUEUE_FLAG_NR,
 };
 
@@ -362,6 +370,7 @@ ELV_IO_QUEUE_FLAG_FNS(sync)
 ELV_IO_QUEUE_FLAG_FNS(wait_request)
 ELV_IO_QUEUE_FLAG_FNS(idle_window)
 ELV_IO_QUEUE_FLAG_FNS(slice_new)
+ELV_IO_QUEUE_FLAG_FNS(wait_busy)
 
 static inline struct io_service_tree *
 io_entity_service_tree(struct io_entity *entity)
@@ -554,6 +563,9 @@ static inline struct io_queue *elv_looku
 extern ssize_t elv_slice_idle_show(struct request_queue *q, char *name);
 extern ssize_t elv_slice_idle_store(struct request_queue *q, const char *name,
 						size_t count);
+extern ssize_t elv_fairness_show(struct request_queue *q, char *name);
+extern ssize_t elv_fairness_store(struct request_queue *q, const char *name,
+						size_t count);
 
 /* Functions used by elevator.c */
 extern int elv_init_fq_data(struct request_queue *q, struct elevator_queue *e);
block/elevator-fq.c +75 -10

143 lines not on this page · raw message

block/blk-sysfs.c +7 -0
Index: linux1/block/blk-sysfs.c
===================================================================
@@ -282,6 +282,12 @@ static struct queue_sysfs_entry queue_sl
 	.show = elv_slice_idle_show,
 	.store = elv_slice_idle_store,
 };
+
+static struct queue_sysfs_entry queue_fairness_entry = {
+	.attr = {.name = "fairness", .mode = S_IRUGO | S_IWUSR },
+	.show = elv_fairness_show,
+	.store = elv_fairness_store,
+};
 #endif
 static struct attribute *default_attrs[] = {
 	&queue_requests_entry.attr,
@@ -296,6 +302,7 @@ static struct attribute *default_attrs[]
 	&queue_iostats_entry.attr,
 #ifdef CONFIG_ELV_FAIR_QUEUING
 	&queue_slice_idle_entry.attr,
+	&queue_fairness_entry.attr,
 #endif
 	NULL,
 };

A message that is not in the archive (sent off-list or not yet received): [email protected]

#75 lore raw

Re: [PATCH 01/10] Documentation

Quoted 69 lines · Vivek Goyal
Quoted 27 lines · Gui Jianfeng
Quoted 19 lines · Vivek Goyal
Quoted 4 lines

Hi Vivek,

I would be interested in knowing if these are the results expected?

Hi Dhaval,

Good question. Keeping current expectation in mind, yes these are expected results. To begin with, current expectations are that try to emulate cfq behavior and the kind of service differentiation we get between threads of different priority, same kind of service differentiation we should get from different cgroups.

Having said that, in theory a more accurate estimate should be amount of actual disk time a queue/cgroup got. I have put a tracing message to keep track of total service received by a queue. If you run "blktrace" then you can see that. Ideally, total service received by two threads over a period of time should be in same proportion as their cgroup weights.

Quoted 7 lines

It will not be easy to achive it given the constraints we have got in terms of how to accurately we can account for disk time actually used by a queue in certain situations. So to begin with I am targetting that try to meet same kind of service differentation between cgroups as cfq provides between threads and then slowly refine it to see how close one can come to get accurate numbers in terms of "total_serivce" received by each queue.

Quoted 4 lines

Hi Vivek,

I simply tested with blktrace opened. I create two groups and set ioprio 4 and 7 respectively(the corresponding weight should 4:1, right?),

Hi Gui,

Thanks for testing. You are right about weight proportions.

Quoted 6 lines

and
start two dd concurrently. UUIC, Ideally, the proportion of service two dd got should be 4:1 in a period of time when they are running. I extract *served* value from blktrace output and sum them up. I found the proportion of the sum of *served* value is about 1.7:1
Am i missing something?

Actually getting the service proportion in same ratio as weight proportion is quite hard for sync queues. The biggest issue is that many a times sync queues are not continuously backlogged and they do some IO and then dispatch a next round of requests.

Most of the time idling seems to be the solution for giving an impression that sync queue is continuously backlogged but it also has potential to reduce throughput on faster hardware.

Anyway, can you please send me your complete blkparse output. There are many a places where code has been designed to favor throughput than fairness. Looking at your blkparse output, will give me better idea what's the issue in your setup.

Also please try the attached patch. I have experimented with waiting for new request to come before sync queue is expired. It helps me in getting the fairness numbers at least with noop on non-queueing rotational media.

I also have introduced a new tunable "fairness". Above code will kick in only if this variable is set to 1. Many a places where we favor throughput over fairness, I plan to use this variable as condition to let user decide whether to choose fairness over throughput. I am not sure at how many places it really makes sense, but it atleast gives us something to play and compare the throughput in two cases.

Quoted 3 lines

This patch applies on my current tree after removing tomost patceh "anticipatory scheduling changes". My code has changed a bit since the posting, so you might have to message this patch a bit.

Hi Vivek,

This time I run two dd with pure sync read like this:
dd if=/mnt/500M.1 of=/dev/null, and the proportion of service got by each is very close to the proportion of their weight.
Previously, I run concurrent dd like this:
dd if=/mnt/500M.1 of=/mnt/500M.2

I'd like to try this patch out.

Quoted 263 lines

Thanks
Vivek

DESC
io-controller: idle for sometime on sync queue before expiring it EDESC

o When a sync queue expires, in many cases it might be empty and then it will be deleted from the active tree. This will lead to a scenario where out of two competing queues, only one is on the tree and when a new queue is selected, vtime jump takes place and we don't see services provided in proportion to weight.

o In general this is a fundamental problem with fairness of sync queues where queues are not continuously backlogged. Looks like idling is only solution to make sure such kind of queues can get some decent amount of disk bandwidth in the face of competion from continusouly backlogged queues. But excessive idling has potential to reduce performance on SSD and disks with commnad queuing.

o This patch experiments with waiting for next request to come before a queue is expired after it has consumed its time slice. This can ensure more accurate fairness numbers in some cases.

o Introduced a tunable "fairness". If set, io-controller will put more focus on getting fairness right than getting throughput right.

---
 block/blk-sysfs.c   |    7 ++++
 block/elevator-fq.c |   85 +++++++++++++++++++++++++++++++++++++++++++++-------
 block/elevator-fq.h |   12 +++++++
 3 files changed, 94 insertions(+), 10 deletions(-)

Index: linux1/block/elevator-fq.h
===================================================================

block/elevator-fq.h +12 -0
@@ -318,6 +318,13 @@ struct elv_fq_data {
 	unsigned long long rate_sampling_start; /*sampling window start jifies*/
 	/* number of sectors finished io during current sampling window */
 	unsigned long rate_sectors_current;
+
+	/*
+	 * If set to 1, will disable many optimizations done for boost
+	 * throughput and focus more on providing fairness for sync
+	 * queues.
+	 */
+	int fairness;
 };
 
 extern int elv_slice_idle;
@@ -340,6 +347,7 @@ enum elv_queue_state_flags {
 	ELV_QUEUE_FLAG_idle_window,	  /* elevator slice idling enabled */
 	ELV_QUEUE_FLAG_wait_request,	  /* waiting for a request */
 	ELV_QUEUE_FLAG_slice_new,	  /* no requests dispatched in slice */
+	ELV_QUEUE_FLAG_wait_busy,	  /* wait for this queue to get busy */
 	ELV_QUEUE_FLAG_NR,
 };
 
@@ -362,6 +370,7 @@ ELV_IO_QUEUE_FLAG_FNS(sync)
 ELV_IO_QUEUE_FLAG_FNS(wait_request)
 ELV_IO_QUEUE_FLAG_FNS(idle_window)
 ELV_IO_QUEUE_FLAG_FNS(slice_new)
+ELV_IO_QUEUE_FLAG_FNS(wait_busy)
 
 static inline struct io_service_tree *
 io_entity_service_tree(struct io_entity *entity)
@@ -554,6 +563,9 @@ static inline struct io_queue *elv_looku
 extern ssize_t elv_slice_idle_show(struct request_queue *q, char *name);
 extern ssize_t elv_slice_idle_store(struct request_queue *q, const char *name,
 						size_t count);
+extern ssize_t elv_fairness_show(struct request_queue *q, char *name);
+extern ssize_t elv_fairness_store(struct request_queue *q, const char *name,
+						size_t count);
 
 /* Functions used by elevator.c */
 extern int elv_init_fq_data(struct request_queue *q, struct elevator_queue *e);

Index: linux1/block/elevator-fq.c
===================================================================

block/elevator-fq.c +75 -10
@@ -1837,6 +1837,44 @@ void elv_ioq_served(struct io_queue *ioq
 			ioq->total_service);
 }
 
+/* Functions to show and store fairness value through sysfs */
+ssize_t elv_fairness_show(struct request_queue *q, char *name)
+{
+	struct elv_fq_data *efqd;
+	unsigned int data;
+	unsigned long flags;
+
+	spin_lock_irqsave(q->queue_lock, flags);
+	efqd = &q->elevator->efqd;
+	data = efqd->fairness;
+	spin_unlock_irqrestore(q->queue_lock, flags);
+	return sprintf(name, "%d\n", data);
+}
+
+ssize_t elv_fairness_store(struct request_queue *q, const char *name,
+			  size_t count)
+{
+	struct elv_fq_data *efqd;
+	unsigned int data;
+	unsigned long flags;
+
+	char *p = (char *)name;
+
+	data = simple_strtoul(p, &p, 10);
+
+	if (data < 0)
+		data = 0;
+	else if (data > INT_MAX)
+		data = INT_MAX;
+
+	spin_lock_irqsave(q->queue_lock, flags);
+	efqd = &q->elevator->efqd;
+	efqd->fairness = data;
+	spin_unlock_irqrestore(q->queue_lock, flags);
+
+	return count;
+}
+
 /* Functions to show and store elv_idle_slice value through sysfs */
 ssize_t elv_slice_idle_show(struct request_queue *q, char *name)
 {
@@ -2263,10 +2301,11 @@ void __elv_ioq_slice_expired(struct requ
 	assert_spin_locked(q->queue_lock);
 	elv_log_ioq(efqd, ioq, "slice expired upd=%d", budget_update);
 
-	if (elv_ioq_wait_request(ioq))
+	if (elv_ioq_wait_request(ioq) || elv_ioq_wait_busy(ioq))
 		del_timer(&efqd->idle_slice_timer);
 
 	elv_clear_ioq_wait_request(ioq);
+	elv_clear_ioq_wait_busy(ioq);
 
 	/*
 	 * if ioq->slice_end = 0, that means a queue was expired before first
@@ -2482,8 +2521,9 @@ void elv_ioq_request_add(struct request_
 		 * immediately and flag that we must not expire this queue
 		 * just now
 		 */
-		if (elv_ioq_wait_request(ioq)) {
+		if (elv_ioq_wait_request(ioq) || elv_ioq_wait_busy(ioq)) {
 			del_timer(&efqd->idle_slice_timer);
+			elv_clear_ioq_wait_busy(ioq);
 			blk_start_queueing(q);
 		}
 	} else if (elv_should_preempt(q, ioq, rq)) {
@@ -2519,6 +2559,9 @@ void elv_idle_slice_timer(unsigned long 
 
 	if (ioq) {
 
+		if (elv_ioq_wait_busy(ioq))
+			goto expire;
+
 		/*
 		 * expired
 		 */
@@ -2546,7 +2589,7 @@ out_cont:
 	spin_unlock_irqrestore(q->queue_lock, flags);
 }
 
-void elv_ioq_arm_slice_timer(struct request_queue *q)
+void elv_ioq_arm_slice_timer(struct request_queue *q, int wait_for_busy)
 {
 	struct elv_fq_data *efqd = &q->elevator->efqd;
 	struct io_queue *ioq = elv_active_ioq(q->elevator);
@@ -2563,15 +2606,27 @@ void elv_ioq_arm_slice_timer(struct requ
 		return;
 
 	/*
-	 * still requests with the driver, don't idle
+	 * idle is disabled, either manually or by past process history
 	 */
-	if (efqd->rq_in_driver)
+	if (!efqd->elv_slice_idle || !elv_ioq_idle_window(ioq))
 		return;
 
 	/*
-	 * idle is disabled, either manually or by past process history
+	 * This queue has consumed its time slice. We are waiting only for
+	 * it to become busy before we select next queue for dispatch.
 	 */
-	if (!efqd->elv_slice_idle || !elv_ioq_idle_window(ioq))
+	if (efqd->fairness && wait_for_busy) {
+		elv_mark_ioq_wait_busy(ioq);
+		sl = efqd->elv_slice_idle;
+		mod_timer(&efqd->idle_slice_timer, jiffies + sl);
+		elv_log(efqd, "arm idle: %lu wait busy=1", sl);
+		return;
+	}
+
+	/*
+	 * still requests with the driver, don't idle
+	 */
+	if (efqd->rq_in_driver)
 		return;
 
 	/*
@@ -2628,6 +2683,12 @@ void *elv_fq_select_ioq(struct request_q
 		}
 	}
 
+	/* We are waiting for this queue to become busy before it expires.*/
+	if (efqd->fairness && elv_ioq_wait_busy(ioq)) {
+		ioq = NULL;
+		goto keep_queue;
+	}
+
 	/*
 	 * The active queue has run out of time, expire it and select new.
 	 */
@@ -2802,10 +2863,14 @@ void elv_ioq_completed_request(struct re
 			elv_ioq_set_prio_slice(q, ioq);
 			elv_clear_ioq_slice_new(ioq);
 		}
-		if (elv_ioq_slice_used(ioq) || elv_ioq_class_idle(ioq))
+		if (elv_ioq_class_idle(ioq))
 			elv_ioq_slice_expired(q, 1);
-		else if (sync && !ioq->nr_queued)
-			elv_ioq_arm_slice_timer(q);
+		else if (sync && !ioq->nr_queued) {
+			if (elv_ioq_slice_used(ioq))
+				elv_ioq_arm_slice_timer(q, 1);
+			else
+				elv_ioq_arm_slice_timer(q, 0);
+		}
 	}
 
 	if (!efqd->rq_in_driver)

Index: linux1/block/blk-sysfs.c
===================================================================

block/blk-sysfs.c +7 -0
@@ -282,6 +282,12 @@ static struct queue_sysfs_entry queue_sl
 	.show = elv_slice_idle_show,
 	.store = elv_slice_idle_store,
 };
+
+static struct queue_sysfs_entry queue_fairness_entry = {
+	.attr = {.name = "fairness", .mode = S_IRUGO | S_IWUSR },
+	.show = elv_fairness_show,
+	.store = elv_fairness_store,
+};
 #endif
 static struct attribute *default_attrs[] = {
 	&queue_requests_entry.attr,
@@ -296,6 +302,7 @@ static struct attribute *default_attrs[]
 	&queue_iostats_entry.attr,
 #ifdef CONFIG_ELV_FAIR_QUEUING
 	&queue_slice_idle_entry.attr,
+	&queue_fairness_entry.attr,
 #endif
 	NULL,
 };
Signature · 2 lines
Regards
Gui Jianfeng
#76 lore raw

Re: [PATCH 01/10] Documentation

In reply to a message not in the archive
Quoted 96 lines · Vivek Goyal
Quoted 29 lines · Gui Jianfeng
Quoted 20 lines · Vivek Goyal
Quoted 4 lines

Hi Vivek,

I would be interested in knowing if these are the results expected?

Hi Dhaval,

Good question. Keeping current expectation in mind, yes these are expected results. To begin with, current expectations are that try to emulate cfq behavior and the kind of service differentiation we get between threads of different priority, same kind of service differentiation we should get from different cgroups.

Having said that, in theory a more accurate estimate should be amount of actual disk time a queue/cgroup got. I have put a tracing message to keep track of total service received by a queue. If you run "blktrace" then you can see that. Ideally, total service received by two threads over a period of time should be in same proportion as their cgroup weights.

Quoted 7 lines

It will not be easy to achive it given the constraints we have got in terms of how to accurately we can account for disk time actually used by a queue in certain situations. So to begin with I am targetting that try to meet same kind of service differentation between cgroups as cfq provides between threads and then slowly refine it to see how close one can come to get accurate numbers in terms of "total_serivce" received by each queue.

Quoted 4 lines

  Hi Vivek,

  I simply tested with blktrace opened. I create two groups and set ioprio 4 and 7 respectively(the corresponding weight should 4:1, right?),

Hi Gui,

Thanks for testing. You are right about weight proportions.

Quoted 6 lines

and
  start two dd concurrently. UUIC, Ideally, the proportion of service two dd got should be 4:1 in a period of time when they are running. I extract *served* value from blktrace output and sum them up. I found the proportion of the sum of *served* value is about 1.7:1
  Am i missing something?

Actually getting the service proportion in same ratio as weight proportion is quite hard for sync queues. The biggest issue is that many a times sync queues are not continuously backlogged and they do some IO and then dispatch a next round of requests.

Most of the time idling seems to be the solution for giving an impression that sync queue is continuously backlogged but it also has potential to reduce throughput on faster hardware.

Anyway, can you please send me your complete blkparse output. There are many a places where code has been designed to favor throughput than fairness. Looking at your blkparse output, will give me better idea what's the issue in your setup.

Also please try the attached patch. I have experimented with waiting for new request to come before sync queue is expired. It helps me in getting the fairness numbers at least with noop on non-queueing rotational media.

I also have introduced a new tunable "fairness". Above code will kick in only if this variable is set to 1. Many a places where we favor throughput over fairness, I plan to use this variable as condition to let user decide whether to choose fairness over throughput. I am not sure at how many places it really makes sense, but it atleast gives us something to play and compare the throughput in two cases.

This patch applies on my current tree after removing tomost patceh "anticipatory scheduling changes". My code has changed a bit since the posting, so you might have to message this patch a bit.

Thanks
Vivek

DESC
io-controller: idle for sometime on sync queue before expiring it EDESC

o When a sync queue expires, in many cases it might be empty and then it will be deleted from the active tree. This will lead to a scenario where out of two competing queues, only one is on the tree and when a new queue is selected, vtime jump takes place and we don't see services provided in proportion to weight.

o In general this is a fundamental problem with fairness of sync queues where queues are not continuously backlogged. Looks like idling is only solution to make sure such kind of queues can get some decent amount of disk bandwidth in the face of competion from continusouly backlogged queues. But excessive idling has potential to reduce performance on SSD and disks with commnad queuing.

Quoted 3 lines

o This patch experiments with waiting for next request to come before a queue is expired after it has consumed its time slice. This can ensure more accurate fairness numbers in some cases.

Vivek, have you introduced this option just to play with it, or you are planning to make it a part of the patch set. Waiting for a new request to come before expiring time slice sounds problematic.

Quoted 236 lines

o Introduced a tunable "fairness". If set, io-controller will put more focus on getting fairness right than getting throughput right.

---
 block/blk-sysfs.c   |    7 ++++
 block/elevator-fq.c |   85 +++++++++++++++++++++++++++++++++++++++++++++------- block/elevator-fq.h |   12 +++++++

 3 files changed, 94 insertions(+), 10 deletions(-)

Index: linux1/block/elevator-fq.h
===================================================================

block/elevator-fq.h  2009-03-18 17:34:53.000000000 -0400 renamed from block/elevator-fq.h     2009-03-18 17:34:46.000000000 -0400 +0 -0
@@ -318,6 +318,13 @@ struct elv_fq_data {

       unsigned long long rate_sampling_start; /*sampling window start jifies*/ /* number of sectors finished io during current sampling window */ unsigned long rate_sectors_current;
+
+       /*
+        * If set to 1, will disable many optimizations done for boost
+        * throughput and focus more on providing fairness for sync
+        * queues.
+        */
+       int fairness;
 };

 extern int elv_slice_idle;

block/elevator-fq.h  2009-03-18 17:34:53.000000000 -0400 renamed from block/elevator-fq.h     2009-03-18 17:34:46.000000000 -0400 +0 -0
@@ -340,6 +347,7 @@ enum elv_queue_state_flags {

       ELV_QUEUE_FLAG_idle_window,       /* elevator slice idling enabled */ ELV_QUEUE_FLAG_wait_request,      /* waiting for a request */ ELV_QUEUE_FLAG_slice_new,         /* no requests dispatched in slice */
+       ELV_QUEUE_FLAG_wait_busy,         /* wait for this queue to get busy */ ELV_QUEUE_FLAG_NR,
 };

block/elevator-fq.h  2009-03-18 17:34:53.000000000 -0400 renamed from block/elevator-fq.h     2009-03-18 17:34:46.000000000 -0400 +1 -0
@@ -362,6 +370,7 @@ ELV_IO_QUEUE_FLAG_FNS(sync)
  ELV_IO_QUEUE_FLAG_FNS(wait_request)
  ELV_IO_QUEUE_FLAG_FNS(idle_window)
  ELV_IO_QUEUE_FLAG_FNS(slice_new)
+ELV_IO_QUEUE_FLAG_FNS(wait_busy)
 

 static inline struct io_service_tree *
 io_entity_service_tree(struct io_entity *entity)

block/elevator-fq.h  2009-03-18 17:34:53.000000000 -0400 renamed from block/elevator-fq.h     2009-03-18 17:34:46.000000000 -0400 +3 -0
@@ -554,6 +563,9 @@ static inline struct io_queue *elv_looku
  extern ssize_t elv_slice_idle_show(struct request_queue *q, char *name);
  extern ssize_t elv_slice_idle_store(struct request_queue *q, const char *name,
                                                size_t count);
+extern ssize_t elv_fairness_show(struct request_queue *q, char *name);
+extern ssize_t elv_fairness_store(struct request_queue *q, const char *name,
+                                               size_t count);
 
  /* Functions used by elevator.c */
  extern int elv_init_fq_data(struct request_queue *q, struct elevator_queue *e);

Index: linux1/block/elevator-fq.c
===================================================================

block/elevator-fq.c  2009-03-18 17:34:53.000000000 -0400 renamed from block/elevator-fq.c     2009-03-18 17:34:46.000000000 -0400 +75 -10
@@ -1837,6 +1837,44 @@ void elv_ioq_served(struct io_queue *ioq
                        ioq->total_service);
  }
 
+/* Functions to show and store fairness value through sysfs */
+ssize_t elv_fairness_show(struct request_queue *q, char *name)
+{
+       struct elv_fq_data *efqd;
+       unsigned int data;
+       unsigned long flags;
+
+       spin_lock_irqsave(q->queue_lock, flags);
+       efqd = &q->elevator->efqd;
+       data = efqd->fairness;
+       spin_unlock_irqrestore(q->queue_lock, flags);
+       return sprintf(name, "%d\n", data);
+}
+
+ssize_t elv_fairness_store(struct request_queue *q, const char *name,
+                         size_t count)
+{
+       struct elv_fq_data *efqd;
+       unsigned int data;
+       unsigned long flags;
+
+       char *p = (char *)name;
+
+       data = simple_strtoul(p, &p, 10);
+
+       if (data < 0)
+               data = 0;
+       else if (data > INT_MAX)
+               data = INT_MAX;
+
+       spin_lock_irqsave(q->queue_lock, flags);
+       efqd = &q->elevator->efqd;
+       efqd->fairness = data;
+       spin_unlock_irqrestore(q->queue_lock, flags);
+
+       return count;
+}
+
  /* Functions to show and store elv_idle_slice value through sysfs */
  ssize_t elv_slice_idle_show(struct request_queue *q, char *name)
  {
@@ -2263,10 +2301,11 @@ void __elv_ioq_slice_expired(struct requ
        assert_spin_locked(q->queue_lock);
        elv_log_ioq(efqd, ioq, "slice expired upd=%d", budget_update);
 
-       if (elv_ioq_wait_request(ioq))
+       if (elv_ioq_wait_request(ioq) || elv_ioq_wait_busy(ioq))
                del_timer(&efqd->idle_slice_timer);
 
        elv_clear_ioq_wait_request(ioq);
+       elv_clear_ioq_wait_busy(ioq);
 
        /*
         * if ioq->slice_end = 0, that means a queue was expired before first
@@ -2482,8 +2521,9 @@ void elv_ioq_request_add(struct request_
                 * immediately and flag that we must not expire this queue
                 * just now
                 */
-               if (elv_ioq_wait_request(ioq)) {
+               if (elv_ioq_wait_request(ioq) || elv_ioq_wait_busy(ioq)) {
                        del_timer(&efqd->idle_slice_timer);
+                       elv_clear_ioq_wait_busy(ioq);
                        blk_start_queueing(q);
                }
        } else if (elv_should_preempt(q, ioq, rq)) {
@@ -2519,6 +2559,9 @@ void elv_idle_slice_timer(unsigned long
 
        if (ioq) {
 
+               if (elv_ioq_wait_busy(ioq))
+                       goto expire;
+
                /*
                 * expired
                 */
@@ -2546,7 +2589,7 @@ out_cont:
        spin_unlock_irqrestore(q->queue_lock, flags);
  }
 
-void elv_ioq_arm_slice_timer(struct request_queue *q)
+void elv_ioq_arm_slice_timer(struct request_queue *q, int wait_for_busy)
  {
        struct elv_fq_data *efqd = &q->elevator->efqd;
        struct io_queue *ioq = elv_active_ioq(q->elevator);
@@ -2563,15 +2606,27 @@ void elv_ioq_arm_slice_timer(struct requ
                return;
 
        /*
-        * still requests with the driver, don't idle
+        * idle is disabled, either manually or by past process history
         */
-       if (efqd->rq_in_driver)
+       if (!efqd->elv_slice_idle || !elv_ioq_idle_window(ioq))
                return;
 
        /*
-        * idle is disabled, either manually or by past process history
+        * This queue has consumed its time slice. We are waiting only for
+        * it to become busy before we select next queue for dispatch.
         */
-       if (!efqd->elv_slice_idle || !elv_ioq_idle_window(ioq))
+       if (efqd->fairness && wait_for_busy) {
+               elv_mark_ioq_wait_busy(ioq);
+               sl = efqd->elv_slice_idle;
+               mod_timer(&efqd->idle_slice_timer, jiffies + sl);
+               elv_log(efqd, "arm idle: %lu wait busy=1", sl);
+               return;
+       }
+
+       /*
+        * still requests with the driver, don't idle
+        */
+       if (efqd->rq_in_driver)
                return;
 
        /*
@@ -2628,6 +2683,12 @@ void *elv_fq_select_ioq(struct request_q
                }
        }
 
+       /* We are waiting for this queue to become busy before it expires.*/
+       if (efqd->fairness && elv_ioq_wait_busy(ioq)) {
+               ioq = NULL;
+               goto keep_queue;
+       }
+
        /*
         * The active queue has run out of time, expire it and select new.
         */
@@ -2802,10 +2863,14 @@ void elv_ioq_completed_request(struct re
                        elv_ioq_set_prio_slice(q, ioq);
                        elv_clear_ioq_slice_new(ioq);
                }
-               if (elv_ioq_slice_used(ioq) || elv_ioq_class_idle(ioq))
+               if (elv_ioq_class_idle(ioq))
                        elv_ioq_slice_expired(q, 1);
-               else if (sync && !ioq->nr_queued)
-                       elv_ioq_arm_slice_timer(q);
+               else if (sync && !ioq->nr_queued) {
+                       if (elv_ioq_slice_used(ioq))
+                               elv_ioq_arm_slice_timer(q, 1);
+                       else
+                               elv_ioq_arm_slice_timer(q, 0);
+               }
        }
 
        if (!efqd->rq_in_driver)

Index: linux1/block/blk-sysfs.c
===================================================================

block/blk-sysfs.c    2009-03-18 17:34:53.000000000 -0400 renamed from block/blk-sysfs.c       2009-03-18 17:34:28.000000000 -0400 +7 -0
@@ -282,6 +282,12 @@ static struct queue_sysfs_entry queue_sl
        .show = elv_slice_idle_show,
        .store = elv_slice_idle_store,
  };
+
+static struct queue_sysfs_entry queue_fairness_entry = {
+       .attr = {.name = "fairness", .mode = S_IRUGO | S_IWUSR },
+       .show = elv_fairness_show,
+       .store = elv_fairness_store,
+};
  #endif
  static struct attribute *default_attrs[] = {
        &queue_requests_entry.attr,
@@ -296,6 +302,7 @@ static struct attribute *default_attrs[]
        &queue_iostats_entry.attr,
  #ifdef CONFIG_ELV_FAIR_QUEUING
        &queue_slice_idle_entry.attr,
+       &queue_fairness_entry.attr,
  #endif
        NULL,
  };
#77 lore raw

Re: [PATCH 01/10] Documentation

In reply to Vivek Goyal
Quoted 69 lines · Vivek Goyal
Quoted 27 lines · Gui Jianfeng
Quoted 19 lines · Vivek Goyal
Quoted 4 lines

Hi Vivek,

I would be interested in knowing if these are the results expected?

Hi Dhaval,

Good question. Keeping current expectation in mind, yes these are expected results. To begin with, current expectations are that try to emulate cfq behavior and the kind of service differentiation we get between threads of different priority, same kind of service differentiation we should get from different cgroups.

Having said that, in theory a more accurate estimate should be amount of actual disk time a queue/cgroup got. I have put a tracing message to keep track of total service received by a queue. If you run "blktrace" then you can see that. Ideally, total service received by two threads over a period of time should be in same proportion as their cgroup weights.

Quoted 7 lines

It will not be easy to achive it given the constraints we have got in terms of how to accurately we can account for disk time actually used by a queue in certain situations. So to begin with I am targetting that try to meet same kind of service differentation between cgroups as cfq provides between threads and then slowly refine it to see how close one can come to get accurate numbers in terms of "total_serivce" received by each queue.

Quoted 4 lines

Hi Vivek,

I simply tested with blktrace opened. I create two groups and set ioprio 4 and 7 respectively(the corresponding weight should 4:1, right?),

Hi Gui,

Thanks for testing. You are right about weight proportions.

Quoted 6 lines

and
start two dd concurrently. UUIC, Ideally, the proportion of service two dd got should be 4:1 in a period of time when they are running. I extract *served* value from blktrace output and sum them up. I found the proportion of the sum of *served* value is about 1.7:1
Am i missing something?

Actually getting the service proportion in same ratio as weight proportion is quite hard for sync queues. The biggest issue is that many a times sync queues are not continuously backlogged and they do some IO and then dispatch a next round of requests.

Most of the time idling seems to be the solution for giving an impression that sync queue is continuously backlogged but it also has potential to reduce throughput on faster hardware.

Anyway, can you please send me your complete blkparse output. There are many a places where code has been designed to favor throughput than fairness. Looking at your blkparse output, will give me better idea what's the issue in your setup.

Also please try the attached patch. I have experimented with waiting for new request to come before sync queue is expired. It helps me in getting the fairness numbers at least with noop on non-queueing rotational media.

I also have introduced a new tunable "fairness". Above code will kick in only if this variable is set to 1. Many a places where we favor throughput over fairness, I plan to use this variable as condition to let user decide whether to choose fairness over throughput. I am not sure at how many places it really makes sense, but it atleast gives us something to play and compare the throughput in two cases.

Quoted 3 lines

This patch applies on my current tree after removing tomost patceh "anticipatory scheduling changes". My code has changed a bit since the posting, so you might have to message this patch a bit.

Hi Vivek,

This time I run two dd with pure sync read like this:
dd if=/mnt/500M.1 of=/dev/null, and the proportion of service got by each is very close to the proportion of their weight.
Previously, I run concurrent dd like this:
dd if=/mnt/500M.1 of=/mnt/500M.2

I'd like to try this patch out.

Quoted 263 lines

Thanks
Vivek

DESC
io-controller: idle for sometime on sync queue before expiring it EDESC

o When a sync queue expires, in many cases it might be empty and then it will be deleted from the active tree. This will lead to a scenario where out of two competing queues, only one is on the tree and when a new queue is selected, vtime jump takes place and we don't see services provided in proportion to weight.

o In general this is a fundamental problem with fairness of sync queues where queues are not continuously backlogged. Looks like idling is only solution to make sure such kind of queues can get some decent amount of disk bandwidth in the face of competion from continusouly backlogged queues. But excessive idling has potential to reduce performance on SSD and disks with commnad queuing.

o This patch experiments with waiting for next request to come before a queue is expired after it has consumed its time slice. This can ensure more accurate fairness numbers in some cases.

o Introduced a tunable "fairness". If set, io-controller will put more focus on getting fairness right than getting throughput right.

---
 block/blk-sysfs.c   |    7 ++++
 block/elevator-fq.c |   85 +++++++++++++++++++++++++++++++++++++++++++++-------
 block/elevator-fq.h |   12 +++++++
 3 files changed, 94 insertions(+), 10 deletions(-)

Index: linux1/block/elevator-fq.h
===================================================================

block/elevator-fq.h +12 -0
@@ -318,6 +318,13 @@ struct elv_fq_data {
 	unsigned long long rate_sampling_start; /*sampling window start jifies*/
 	/* number of sectors finished io during current sampling window */
 	unsigned long rate_sectors_current;
+
+	/*
+	 * If set to 1, will disable many optimizations done for boost
+	 * throughput and focus more on providing fairness for sync
+	 * queues.
+	 */
+	int fairness;
 };
 
 extern int elv_slice_idle;
@@ -340,6 +347,7 @@ enum elv_queue_state_flags {
 	ELV_QUEUE_FLAG_idle_window,	  /* elevator slice idling enabled */
 	ELV_QUEUE_FLAG_wait_request,	  /* waiting for a request */
 	ELV_QUEUE_FLAG_slice_new,	  /* no requests dispatched in slice */
+	ELV_QUEUE_FLAG_wait_busy,	  /* wait for this queue to get busy */
 	ELV_QUEUE_FLAG_NR,
 };
 
@@ -362,6 +370,7 @@ ELV_IO_QUEUE_FLAG_FNS(sync)
 ELV_IO_QUEUE_FLAG_FNS(wait_request)
 ELV_IO_QUEUE_FLAG_FNS(idle_window)
 ELV_IO_QUEUE_FLAG_FNS(slice_new)
+ELV_IO_QUEUE_FLAG_FNS(wait_busy)
 
 static inline struct io_service_tree *
 io_entity_service_tree(struct io_entity *entity)
@@ -554,6 +563,9 @@ static inline struct io_queue *elv_looku
 extern ssize_t elv_slice_idle_show(struct request_queue *q, char *name);
 extern ssize_t elv_slice_idle_store(struct request_queue *q, const char *name,
 						size_t count);
+extern ssize_t elv_fairness_show(struct request_queue *q, char *name);
+extern ssize_t elv_fairness_store(struct request_queue *q, const char *name,
+						size_t count);
 
 /* Functions used by elevator.c */
 extern int elv_init_fq_data(struct request_queue *q, struct elevator_queue *e);

Index: linux1/block/elevator-fq.c
===================================================================

block/elevator-fq.c +75 -10
@@ -1837,6 +1837,44 @@ void elv_ioq_served(struct io_queue *ioq
 			ioq->total_service);
 }
 
+/* Functions to show and store fairness value through sysfs */
+ssize_t elv_fairness_show(struct request_queue *q, char *name)
+{
+	struct elv_fq_data *efqd;
+	unsigned int data;
+	unsigned long flags;
+
+	spin_lock_irqsave(q->queue_lock, flags);
+	efqd = &q->elevator->efqd;
+	data = efqd->fairness;
+	spin_unlock_irqrestore(q->queue_lock, flags);
+	return sprintf(name, "%d\n", data);
+}
+
+ssize_t elv_fairness_store(struct request_queue *q, const char *name,
+			  size_t count)
+{
+	struct elv_fq_data *efqd;
+	unsigned int data;
+	unsigned long flags;
+
+	char *p = (char *)name;
+
+	data = simple_strtoul(p, &p, 10);
+
+	if (data < 0)
+		data = 0;
+	else if (data > INT_MAX)
+		data = INT_MAX;
+
+	spin_lock_irqsave(q->queue_lock, flags);
+	efqd = &q->elevator->efqd;
+	efqd->fairness = data;
+	spin_unlock_irqrestore(q->queue_lock, flags);
+
+	return count;
+}
+
 /* Functions to show and store elv_idle_slice value through sysfs */
 ssize_t elv_slice_idle_show(struct request_queue *q, char *name)
 {
@@ -2263,10 +2301,11 @@ void __elv_ioq_slice_expired(struct requ
 	assert_spin_locked(q->queue_lock);
 	elv_log_ioq(efqd, ioq, "slice expired upd=%d", budget_update);
 
-	if (elv_ioq_wait_request(ioq))
+	if (elv_ioq_wait_request(ioq) || elv_ioq_wait_busy(ioq))
 		del_timer(&efqd->idle_slice_timer);
 
 	elv_clear_ioq_wait_request(ioq);
+	elv_clear_ioq_wait_busy(ioq);
 
 	/*
 	 * if ioq->slice_end = 0, that means a queue was expired before first
@@ -2482,8 +2521,9 @@ void elv_ioq_request_add(struct request_
 		 * immediately and flag that we must not expire this queue
 		 * just now
 		 */
-		if (elv_ioq_wait_request(ioq)) {
+		if (elv_ioq_wait_request(ioq) || elv_ioq_wait_busy(ioq)) {
 			del_timer(&efqd->idle_slice_timer);
+			elv_clear_ioq_wait_busy(ioq);
 			blk_start_queueing(q);
 		}
 	} else if (elv_should_preempt(q, ioq, rq)) {
@@ -2519,6 +2559,9 @@ void elv_idle_slice_timer(unsigned long 
 
 	if (ioq) {
 
+		if (elv_ioq_wait_busy(ioq))
+			goto expire;
+
 		/*
 		 * expired
 		 */
@@ -2546,7 +2589,7 @@ out_cont:
 	spin_unlock_irqrestore(q->queue_lock, flags);
 }
 
-void elv_ioq_arm_slice_timer(struct request_queue *q)
+void elv_ioq_arm_slice_timer(struct request_queue *q, int wait_for_busy)
 {
 	struct elv_fq_data *efqd = &q->elevator->efqd;
 	struct io_queue *ioq = elv_active_ioq(q->elevator);
@@ -2563,15 +2606,27 @@ void elv_ioq_arm_slice_timer(struct requ
 		return;
 
 	/*
-	 * still requests with the driver, don't idle
+	 * idle is disabled, either manually or by past process history
 	 */
-	if (efqd->rq_in_driver)
+	if (!efqd->elv_slice_idle || !elv_ioq_idle_window(ioq))
 		return;
 
 	/*
-	 * idle is disabled, either manually or by past process history
+	 * This queue has consumed its time slice. We are waiting only for
+	 * it to become busy before we select next queue for dispatch.
 	 */
-	if (!efqd->elv_slice_idle || !elv_ioq_idle_window(ioq))
+	if (efqd->fairness && wait_for_busy) {
+		elv_mark_ioq_wait_busy(ioq);
+		sl = efqd->elv_slice_idle;
+		mod_timer(&efqd->idle_slice_timer, jiffies + sl);
+		elv_log(efqd, "arm idle: %lu wait busy=1", sl);
+		return;
+	}
+
+	/*
+	 * still requests with the driver, don't idle
+	 */
+	if (efqd->rq_in_driver)
 		return;
 
 	/*
@@ -2628,6 +2683,12 @@ void *elv_fq_select_ioq(struct request_q
 		}
 	}
 
+	/* We are waiting for this queue to become busy before it expires.*/
+	if (efqd->fairness && elv_ioq_wait_busy(ioq)) {
+		ioq = NULL;
+		goto keep_queue;
+	}
+
 	/*
 	 * The active queue has run out of time, expire it and select new.
 	 */
@@ -2802,10 +2863,14 @@ void elv_ioq_completed_request(struct re
 			elv_ioq_set_prio_slice(q, ioq);
 			elv_clear_ioq_slice_new(ioq);
 		}
-		if (elv_ioq_slice_used(ioq) || elv_ioq_class_idle(ioq))
+		if (elv_ioq_class_idle(ioq))
 			elv_ioq_slice_expired(q, 1);
-		else if (sync && !ioq->nr_queued)
-			elv_ioq_arm_slice_timer(q);
+		else if (sync && !ioq->nr_queued) {
+			if (elv_ioq_slice_used(ioq))
+				elv_ioq_arm_slice_timer(q, 1);
+			else
+				elv_ioq_arm_slice_timer(q, 0);
+		}
 	}
 
 	if (!efqd->rq_in_driver)

Index: linux1/block/blk-sysfs.c
===================================================================

block/blk-sysfs.c +7 -0
@@ -282,6 +282,12 @@ static struct queue_sysfs_entry queue_sl
 	.show = elv_slice_idle_show,
 	.store = elv_slice_idle_store,
 };
+
+static struct queue_sysfs_entry queue_fairness_entry = {
+	.attr = {.name = "fairness", .mode = S_IRUGO | S_IWUSR },
+	.show = elv_fairness_show,
+	.store = elv_fairness_store,
+};
 #endif
 static struct attribute *default_attrs[] = {
 	&queue_requests_entry.attr,
@@ -296,6 +302,7 @@ static struct attribute *default_attrs[]
 	&queue_iostats_entry.attr,
 #ifdef CONFIG_ELV_FAIR_QUEUING
 	&queue_slice_idle_entry.attr,
+	&queue_fairness_entry.attr,
 #endif
 	NULL,
 };
Signature · 2 lines
Regards
Gui Jianfeng
#78 lore raw

Re: [PATCH 01/10] Documentation

In reply to Vivek Goyal
Quoted 96 lines · Vivek Goyal
Quoted 29 lines · Gui Jianfeng
Quoted 20 lines · Vivek Goyal
Quoted 4 lines

Hi Vivek,

I would be interested in knowing if these are the results expected?

Hi Dhaval,

Good question. Keeping current expectation in mind, yes these are expected results. To begin with, current expectations are that try to emulate cfq behavior and the kind of service differentiation we get between threads of different priority, same kind of service differentiation we should get from different cgroups.

Having said that, in theory a more accurate estimate should be amount of actual disk time a queue/cgroup got. I have put a tracing message to keep track of total service received by a queue. If you run "blktrace" then you can see that. Ideally, total service received by two threads over a period of time should be in same proportion as their cgroup weights.

Quoted 7 lines

It will not be easy to achive it given the constraints we have got in terms of how to accurately we can account for disk time actually used by a queue in certain situations. So to begin with I am targetting that try to meet same kind of service differentation between cgroups as cfq provides between threads and then slowly refine it to see how close one can come to get accurate numbers in terms of "total_serivce" received by each queue.

Quoted 4 lines

  Hi Vivek,

  I simply tested with blktrace opened. I create two groups and set ioprio 4 and 7 respectively(the corresponding weight should 4:1, right?),

Hi Gui,

Thanks for testing. You are right about weight proportions.

Quoted 6 lines

and
  start two dd concurrently. UUIC, Ideally, the proportion of service two dd got should be 4:1 in a period of time when they are running. I extract *served* value from blktrace output and sum them up. I found the proportion of the sum of *served* value is about 1.7:1
  Am i missing something?

Actually getting the service proportion in same ratio as weight proportion is quite hard for sync queues. The biggest issue is that many a times sync queues are not continuously backlogged and they do some IO and then dispatch a next round of requests.

Most of the time idling seems to be the solution for giving an impression that sync queue is continuously backlogged but it also has potential to reduce throughput on faster hardware.

Anyway, can you please send me your complete blkparse output. There are many a places where code has been designed to favor throughput than fairness. Looking at your blkparse output, will give me better idea what's the issue in your setup.

Also please try the attached patch. I have experimented with waiting for new request to come before sync queue is expired. It helps me in getting the fairness numbers at least with noop on non-queueing rotational media.

I also have introduced a new tunable "fairness". Above code will kick in only if this variable is set to 1. Many a places where we favor throughput over fairness, I plan to use this variable as condition to let user decide whether to choose fairness over throughput. I am not sure at how many places it really makes sense, but it atleast gives us something to play and compare the throughput in two cases.

This patch applies on my current tree after removing tomost patceh "anticipatory scheduling changes". My code has changed a bit since the posting, so you might have to message this patch a bit.

Thanks
Vivek

DESC
io-controller: idle for sometime on sync queue before expiring it EDESC

o When a sync queue expires, in many cases it might be empty and then it will be deleted from the active tree. This will lead to a scenario where out of two competing queues, only one is on the tree and when a new queue is selected, vtime jump takes place and we don't see services provided in proportion to weight.

o In general this is a fundamental problem with fairness of sync queues where queues are not continuously backlogged. Looks like idling is only solution to make sure such kind of queues can get some decent amount of disk bandwidth in the face of competion from continusouly backlogged queues. But excessive idling has potential to reduce performance on SSD and disks with commnad queuing.

Quoted 3 lines

o This patch experiments with waiting for next request to come before a queue is expired after it has consumed its time slice. This can ensure more accurate fairness numbers in some cases.

Vivek, have you introduced this option just to play with it, or you are planning to make it a part of the patch set. Waiting for a new request to come before expiring time slice sounds problematic.

Quoted 236 lines

o Introduced a tunable "fairness". If set, io-controller will put more focus on getting fairness right than getting throughput right.

---
 block/blk-sysfs.c   |    7 ++++
 block/elevator-fq.c |   85 +++++++++++++++++++++++++++++++++++++++++++++------- block/elevator-fq.h |   12 +++++++

 3 files changed, 94 insertions(+), 10 deletions(-)

Index: linux1/block/elevator-fq.h
===================================================================

block/elevator-fq.h  2009-03-18 17:34:53.000000000 -0400 renamed from block/elevator-fq.h     2009-03-18 17:34:46.000000000 -0400 +0 -0
@@ -318,6 +318,13 @@ struct elv_fq_data {

       unsigned long long rate_sampling_start; /*sampling window start jifies*/ /* number of sectors finished io during current sampling window */ unsigned long rate_sectors_current;
+
+       /*
+        * If set to 1, will disable many optimizations done for boost
+        * throughput and focus more on providing fairness for sync
+        * queues.
+        */
+       int fairness;
 };

 extern int elv_slice_idle;

block/elevator-fq.h  2009-03-18 17:34:53.000000000 -0400 renamed from block/elevator-fq.h     2009-03-18 17:34:46.000000000 -0400 +0 -0
@@ -340,6 +347,7 @@ enum elv_queue_state_flags {

       ELV_QUEUE_FLAG_idle_window,       /* elevator slice idling enabled */ ELV_QUEUE_FLAG_wait_request,      /* waiting for a request */ ELV_QUEUE_FLAG_slice_new,         /* no requests dispatched in slice */
+       ELV_QUEUE_FLAG_wait_busy,         /* wait for this queue to get busy */ ELV_QUEUE_FLAG_NR,
 };

block/elevator-fq.h  2009-03-18 17:34:53.000000000 -0400 renamed from block/elevator-fq.h     2009-03-18 17:34:46.000000000 -0400 +1 -0
@@ -362,6 +370,7 @@ ELV_IO_QUEUE_FLAG_FNS(sync)
  ELV_IO_QUEUE_FLAG_FNS(wait_request)
  ELV_IO_QUEUE_FLAG_FNS(idle_window)
  ELV_IO_QUEUE_FLAG_FNS(slice_new)
+ELV_IO_QUEUE_FLAG_FNS(wait_busy)
 

 static inline struct io_service_tree *
 io_entity_service_tree(struct io_entity *entity)

block/elevator-fq.h  2009-03-18 17:34:53.000000000 -0400 renamed from block/elevator-fq.h     2009-03-18 17:34:46.000000000 -0400 +3 -0
@@ -554,6 +563,9 @@ static inline struct io_queue *elv_looku
  extern ssize_t elv_slice_idle_show(struct request_queue *q, char *name);
  extern ssize_t elv_slice_idle_store(struct request_queue *q, const char *name,
                                                size_t count);
+extern ssize_t elv_fairness_show(struct request_queue *q, char *name);
+extern ssize_t elv_fairness_store(struct request_queue *q, const char *name,
+                                               size_t count);
 
  /* Functions used by elevator.c */
  extern int elv_init_fq_data(struct request_queue *q, struct elevator_queue *e);

Index: linux1/block/elevator-fq.c
===================================================================

block/elevator-fq.c  2009-03-18 17:34:53.000000000 -0400 renamed from block/elevator-fq.c     2009-03-18 17:34:46.000000000 -0400 +75 -10
@@ -1837,6 +1837,44 @@ void elv_ioq_served(struct io_queue *ioq
                        ioq->total_service);
  }
 
+/* Functions to show and store fairness value through sysfs */
+ssize_t elv_fairness_show(struct request_queue *q, char *name)
+{
+       struct elv_fq_data *efqd;
+       unsigned int data;
+       unsigned long flags;
+
+       spin_lock_irqsave(q->queue_lock, flags);
+       efqd = &q->elevator->efqd;
+       data = efqd->fairness;
+       spin_unlock_irqrestore(q->queue_lock, flags);
+       return sprintf(name, "%d\n", data);
+}
+
+ssize_t elv_fairness_store(struct request_queue *q, const char *name,
+                         size_t count)
+{
+       struct elv_fq_data *efqd;
+       unsigned int data;
+       unsigned long flags;
+
+       char *p = (char *)name;
+
+       data = simple_strtoul(p, &p, 10);
+
+       if (data < 0)
+               data = 0;
+       else if (data > INT_MAX)
+               data = INT_MAX;
+
+       spin_lock_irqsave(q->queue_lock, flags);
+       efqd = &q->elevator->efqd;
+       efqd->fairness = data;
+       spin_unlock_irqrestore(q->queue_lock, flags);
+
+       return count;
+}
+
  /* Functions to show and store elv_idle_slice value through sysfs */
  ssize_t elv_slice_idle_show(struct request_queue *q, char *name)
  {
@@ -2263,10 +2301,11 @@ void __elv_ioq_slice_expired(struct requ
        assert_spin_locked(q->queue_lock);
        elv_log_ioq(efqd, ioq, "slice expired upd=%d", budget_update);
 
-       if (elv_ioq_wait_request(ioq))
+       if (elv_ioq_wait_request(ioq) || elv_ioq_wait_busy(ioq))
                del_timer(&efqd->idle_slice_timer);
 
        elv_clear_ioq_wait_request(ioq);
+       elv_clear_ioq_wait_busy(ioq);
 
        /*
         * if ioq->slice_end = 0, that means a queue was expired before first
@@ -2482,8 +2521,9 @@ void elv_ioq_request_add(struct request_
                 * immediately and flag that we must not expire this queue
                 * just now
                 */
-               if (elv_ioq_wait_request(ioq)) {
+               if (elv_ioq_wait_request(ioq) || elv_ioq_wait_busy(ioq)) {
                        del_timer(&efqd->idle_slice_timer);
+                       elv_clear_ioq_wait_busy(ioq);
                        blk_start_queueing(q);
                }
        } else if (elv_should_preempt(q, ioq, rq)) {
@@ -2519,6 +2559,9 @@ void elv_idle_slice_timer(unsigned long
 
        if (ioq) {
 
+               if (elv_ioq_wait_busy(ioq))
+                       goto expire;
+
                /*
                 * expired
                 */
@@ -2546,7 +2589,7 @@ out_cont:
        spin_unlock_irqrestore(q->queue_lock, flags);
  }
 
-void elv_ioq_arm_slice_timer(struct request_queue *q)
+void elv_ioq_arm_slice_timer(struct request_queue *q, int wait_for_busy)
  {
        struct elv_fq_data *efqd = &q->elevator->efqd;
        struct io_queue *ioq = elv_active_ioq(q->elevator);
@@ -2563,15 +2606,27 @@ void elv_ioq_arm_slice_timer(struct requ
                return;
 
        /*
-        * still requests with the driver, don't idle
+        * idle is disabled, either manually or by past process history
         */
-       if (efqd->rq_in_driver)
+       if (!efqd->elv_slice_idle || !elv_ioq_idle_window(ioq))
                return;
 
        /*
-        * idle is disabled, either manually or by past process history
+        * This queue has consumed its time slice. We are waiting only for
+        * it to become busy before we select next queue for dispatch.
         */
-       if (!efqd->elv_slice_idle || !elv_ioq_idle_window(ioq))
+       if (efqd->fairness && wait_for_busy) {
+               elv_mark_ioq_wait_busy(ioq);
+               sl = efqd->elv_slice_idle;
+               mod_timer(&efqd->idle_slice_timer, jiffies + sl);
+               elv_log(efqd, "arm idle: %lu wait busy=1", sl);
+               return;
+       }
+
+       /*
+        * still requests with the driver, don't idle
+        */
+       if (efqd->rq_in_driver)
                return;
 
        /*
@@ -2628,6 +2683,12 @@ void *elv_fq_select_ioq(struct request_q
                }
        }
 
+       /* We are waiting for this queue to become busy before it expires.*/
+       if (efqd->fairness && elv_ioq_wait_busy(ioq)) {
+               ioq = NULL;
+               goto keep_queue;
+       }
+
        /*
         * The active queue has run out of time, expire it and select new.
         */
@@ -2802,10 +2863,14 @@ void elv_ioq_completed_request(struct re
                        elv_ioq_set_prio_slice(q, ioq);
                        elv_clear_ioq_slice_new(ioq);
                }
-               if (elv_ioq_slice_used(ioq) || elv_ioq_class_idle(ioq))
+               if (elv_ioq_class_idle(ioq))
                        elv_ioq_slice_expired(q, 1);
-               else if (sync && !ioq->nr_queued)
-                       elv_ioq_arm_slice_timer(q);
+               else if (sync && !ioq->nr_queued) {
+                       if (elv_ioq_slice_used(ioq))
+                               elv_ioq_arm_slice_timer(q, 1);
+                       else
+                               elv_ioq_arm_slice_timer(q, 0);
+               }
        }
 
        if (!efqd->rq_in_driver)

Index: linux1/block/blk-sysfs.c
===================================================================

block/blk-sysfs.c    2009-03-18 17:34:53.000000000 -0400 renamed from block/blk-sysfs.c       2009-03-18 17:34:28.000000000 -0400 +7 -0
@@ -282,6 +282,12 @@ static struct queue_sysfs_entry queue_sl
        .show = elv_slice_idle_show,
        .store = elv_slice_idle_store,
  };
+
+static struct queue_sysfs_entry queue_fairness_entry = {
+       .attr = {.name = "fairness", .mode = S_IRUGO | S_IWUSR },
+       .show = elv_fairness_show,
+       .store = elv_fairness_store,
+};
  #endif
  static struct attribute *default_attrs[] = {
        &queue_requests_entry.attr,
@@ -296,6 +302,7 @@ static struct attribute *default_attrs[]
        &queue_iostats_entry.attr,
  #ifdef CONFIG_ELV_FAIR_QUEUING
        &queue_slice_idle_entry.attr,
+       &queue_fairness_entry.attr,
  #endif
        NULL,
  };
#79 lore raw

Re: [PATCH 01/10] Documentation

On Mon, Mar 23, 2009 at 10:32:41PM -0700, Nauman Rafique wrote:

[..]

Quoted 17 lines
Quoted 17 lines

DESC
io-controller: idle for sometime on sync queue before expiring it EDESC

o When a sync queue expires, in many cases it might be empty and then it will be deleted from the active tree. This will lead to a scenario where out of two competing queues, only one is on the tree and when a new queue is selected, vtime jump takes place and we don't see services provided in proportion to weight.

o In general this is a fundamental problem with fairness of sync queues where queues are not continuously backlogged. Looks like idling is only solution to make sure such kind of queues can get some decent amount of disk bandwidth in the face of competion from continusouly backlogged queues. But excessive idling has potential to reduce performance on SSD and disks with commnad queuing.

Quoted 7 lines
Quoted 3 lines

o This patch experiments with waiting for next request to come before a queue is expired after it has consumed its time slice. This can ensure more accurate fairness numbers in some cases.

Vivek, have you introduced this option just to play with it, or you are planning to make it a part of the patch set. Waiting for a new request to come before expiring time slice sounds problematic.

Why are the issues you forsee with it. This is just an extra 8ms idling on the sync queue that is also if think time of the queue is not high.

We already do idling on sync queues. In this case we are doing an extra idle even if queue has consumed its allocated quota. It helps me get fairness numbers and I have put it under a tunable "fairness". So by default this code will not kick in.

Other possible option could be that when expiring a sync queue, don't remove the queue immediately from the tree and remove it later if there is no request from the queue in 8ms or so. I am not sure with BFQ, is it feasible to do that without creating issues with current implementation. Current implementation was simple, so I stick to it to begin with.

So yes, I am planning to keep it under tunable, unless there are significant issues in doing that.

Thanks
Vivek

Quoted 237 lines
Quoted 237 lines

o Introduced a tunable "fairness". If set, io-controller will put more focus on getting fairness right than getting throughput right.

---
 block/blk-sysfs.c   |    7 ++++
 block/elevator-fq.c |   85 +++++++++++++++++++++++++++++++++++++++++++++------- block/elevator-fq.h |   12 +++++++

 3 files changed, 94 insertions(+), 10 deletions(-)

Index: linux1/block/elevator-fq.h
===================================================================

block/elevator-fq.h  2009-03-18 17:34:53.000000000 -0400 renamed from block/elevator-fq.h     2009-03-18 17:34:46.000000000 -0400 +0 -0
@@ -318,6 +318,13 @@ struct elv_fq_data {

       unsigned long long rate_sampling_start; /*sampling window start jifies*/ /* number of sectors finished io during current sampling window */ unsigned long rate_sectors_current;
+
+       /*
+        * If set to 1, will disable many optimizations done for boost
+        * throughput and focus more on providing fairness for sync
+        * queues.
+        */
+       int fairness;
 };

 extern int elv_slice_idle;

block/elevator-fq.h  2009-03-18 17:34:53.000000000 -0400 renamed from block/elevator-fq.h     2009-03-18 17:34:46.000000000 -0400 +0 -0
@@ -340,6 +347,7 @@ enum elv_queue_state_flags {

       ELV_QUEUE_FLAG_idle_window,       /* elevator slice idling enabled */ ELV_QUEUE_FLAG_wait_request,      /* waiting for a request */ ELV_QUEUE_FLAG_slice_new,         /* no requests dispatched in slice */
+       ELV_QUEUE_FLAG_wait_busy,         /* wait for this queue to get busy */ ELV_QUEUE_FLAG_NR,
 };

block/elevator-fq.h  2009-03-18 17:34:53.000000000 -0400 renamed from block/elevator-fq.h     2009-03-18 17:34:46.000000000 -0400 +1 -0
@@ -362,6 +370,7 @@ ELV_IO_QUEUE_FLAG_FNS(sync)
  ELV_IO_QUEUE_FLAG_FNS(wait_request)
  ELV_IO_QUEUE_FLAG_FNS(idle_window)
  ELV_IO_QUEUE_FLAG_FNS(slice_new)
+ELV_IO_QUEUE_FLAG_FNS(wait_busy)
 

 static inline struct io_service_tree *
 io_entity_service_tree(struct io_entity *entity)

block/elevator-fq.h  2009-03-18 17:34:53.000000000 -0400 renamed from block/elevator-fq.h     2009-03-18 17:34:46.000000000 -0400 +3 -0
@@ -554,6 +563,9 @@ static inline struct io_queue *elv_looku
  extern ssize_t elv_slice_idle_show(struct request_queue *q, char *name);
  extern ssize_t elv_slice_idle_store(struct request_queue *q, const char *name,
                                                size_t count);
+extern ssize_t elv_fairness_show(struct request_queue *q, char *name);
+extern ssize_t elv_fairness_store(struct request_queue *q, const char *name,
+                                               size_t count);
 
  /* Functions used by elevator.c */
  extern int elv_init_fq_data(struct request_queue *q, struct elevator_queue *e);

Index: linux1/block/elevator-fq.c
===================================================================

block/elevator-fq.c  2009-03-18 17:34:53.000000000 -0400 renamed from block/elevator-fq.c     2009-03-18 17:34:46.000000000 -0400 +75 -10
@@ -1837,6 +1837,44 @@ void elv_ioq_served(struct io_queue *ioq
                        ioq->total_service);
  }
 
+/* Functions to show and store fairness value through sysfs */
+ssize_t elv_fairness_show(struct request_queue *q, char *name)
+{
+       struct elv_fq_data *efqd;
+       unsigned int data;
+       unsigned long flags;
+
+       spin_lock_irqsave(q->queue_lock, flags);
+       efqd = &q->elevator->efqd;
+       data = efqd->fairness;
+       spin_unlock_irqrestore(q->queue_lock, flags);
+       return sprintf(name, "%d\n", data);
+}
+
+ssize_t elv_fairness_store(struct request_queue *q, const char *name,
+                         size_t count)
+{
+       struct elv_fq_data *efqd;
+       unsigned int data;
+       unsigned long flags;
+
+       char *p = (char *)name;
+
+       data = simple_strtoul(p, &p, 10);
+
+       if (data < 0)
+               data = 0;
+       else if (data > INT_MAX)
+               data = INT_MAX;
+
+       spin_lock_irqsave(q->queue_lock, flags);
+       efqd = &q->elevator->efqd;
+       efqd->fairness = data;
+       spin_unlock_irqrestore(q->queue_lock, flags);
+
+       return count;
+}
+
  /* Functions to show and store elv_idle_slice value through sysfs */
  ssize_t elv_slice_idle_show(struct request_queue *q, char *name)
  {
@@ -2263,10 +2301,11 @@ void __elv_ioq_slice_expired(struct requ
        assert_spin_locked(q->queue_lock);
        elv_log_ioq(efqd, ioq, "slice expired upd=%d", budget_update);
 
-       if (elv_ioq_wait_request(ioq))
+       if (elv_ioq_wait_request(ioq) || elv_ioq_wait_busy(ioq))
                del_timer(&efqd->idle_slice_timer);
 
        elv_clear_ioq_wait_request(ioq);
+       elv_clear_ioq_wait_busy(ioq);
 
        /*
         * if ioq->slice_end = 0, that means a queue was expired before first
@@ -2482,8 +2521,9 @@ void elv_ioq_request_add(struct request_
                 * immediately and flag that we must not expire this queue
                 * just now
                 */
-               if (elv_ioq_wait_request(ioq)) {
+               if (elv_ioq_wait_request(ioq) || elv_ioq_wait_busy(ioq)) {
                        del_timer(&efqd->idle_slice_timer);
+                       elv_clear_ioq_wait_busy(ioq);
                        blk_start_queueing(q);
                }
        } else if (elv_should_preempt(q, ioq, rq)) {
@@ -2519,6 +2559,9 @@ void elv_idle_slice_timer(unsigned long
 
        if (ioq) {
 
+               if (elv_ioq_wait_busy(ioq))
+                       goto expire;
+
                /*
                 * expired
                 */
@@ -2546,7 +2589,7 @@ out_cont:
        spin_unlock_irqrestore(q->queue_lock, flags);
  }
 
-void elv_ioq_arm_slice_timer(struct request_queue *q)
+void elv_ioq_arm_slice_timer(struct request_queue *q, int wait_for_busy)
  {
        struct elv_fq_data *efqd = &q->elevator->efqd;
        struct io_queue *ioq = elv_active_ioq(q->elevator);
@@ -2563,15 +2606,27 @@ void elv_ioq_arm_slice_timer(struct requ
                return;
 
        /*
-        * still requests with the driver, don't idle
+        * idle is disabled, either manually or by past process history
         */
-       if (efqd->rq_in_driver)
+       if (!efqd->elv_slice_idle || !elv_ioq_idle_window(ioq))
                return;
 
        /*
-        * idle is disabled, either manually or by past process history
+        * This queue has consumed its time slice. We are waiting only for
+        * it to become busy before we select next queue for dispatch.
         */
-       if (!efqd->elv_slice_idle || !elv_ioq_idle_window(ioq))
+       if (efqd->fairness && wait_for_busy) {
+               elv_mark_ioq_wait_busy(ioq);
+               sl = efqd->elv_slice_idle;
+               mod_timer(&efqd->idle_slice_timer, jiffies + sl);
+               elv_log(efqd, "arm idle: %lu wait busy=1", sl);
+               return;
+       }
+
+       /*
+        * still requests with the driver, don't idle
+        */
+       if (efqd->rq_in_driver)
                return;
 
        /*
@@ -2628,6 +2683,12 @@ void *elv_fq_select_ioq(struct request_q
                }
        }
 
+       /* We are waiting for this queue to become busy before it expires.*/
+       if (efqd->fairness && elv_ioq_wait_busy(ioq)) {
+               ioq = NULL;
+               goto keep_queue;
+       }
+
        /*
         * The active queue has run out of time, expire it and select new.
         */
@@ -2802,10 +2863,14 @@ void elv_ioq_completed_request(struct re
                        elv_ioq_set_prio_slice(q, ioq);
                        elv_clear_ioq_slice_new(ioq);
                }
-               if (elv_ioq_slice_used(ioq) || elv_ioq_class_idle(ioq))
+               if (elv_ioq_class_idle(ioq))
                        elv_ioq_slice_expired(q, 1);
-               else if (sync && !ioq->nr_queued)
-                       elv_ioq_arm_slice_timer(q);
+               else if (sync && !ioq->nr_queued) {
+                       if (elv_ioq_slice_used(ioq))
+                               elv_ioq_arm_slice_timer(q, 1);
+                       else
+                               elv_ioq_arm_slice_timer(q, 0);
+               }
        }
 
        if (!efqd->rq_in_driver)

Index: linux1/block/blk-sysfs.c
===================================================================

block/blk-sysfs.c    2009-03-18 17:34:53.000000000 -0400 renamed from block/blk-sysfs.c       2009-03-18 17:34:28.000000000 -0400 +7 -0
@@ -282,6 +282,12 @@ static struct queue_sysfs_entry queue_sl
        .show = elv_slice_idle_show,
        .store = elv_slice_idle_store,
  };
+
+static struct queue_sysfs_entry queue_fairness_entry = {
+       .attr = {.name = "fairness", .mode = S_IRUGO | S_IWUSR },
+       .show = elv_fairness_show,
+       .store = elv_fairness_store,
+};
  #endif
  static struct attribute *default_attrs[] = {
        &queue_requests_entry.attr,
@@ -296,6 +302,7 @@ static struct attribute *default_attrs[]
        &queue_iostats_entry.attr,
  #ifdef CONFIG_ELV_FAIR_QUEUING
        &queue_slice_idle_entry.attr,
+       &queue_fairness_entry.attr,
  #endif
        NULL,
  };

A message that is not in the archive (sent off-list or not yet received): [email protected]

#81 lore raw

Re: [PATCH 01/10] Documentation

Quoted 36 lines · Vivek Goyal

On Mon, Mar 23, 2009 at 10:32:41PM -0700, Nauman Rafique wrote:

[..]

Quoted 17 lines
Quoted 17 lines

DESC
io-controller: idle for sometime on sync queue before expiring it EDESC

o When a sync queue expires, in many cases it might be empty and then áit will be deleted from the active tree. This will lead to a scenario áwhere out of two competing queues, only one is on the tree and when a ánew queue is selected, vtime jump takes place and we don't see services áprovided in proportion to weight.

o In general this is a fundamental problem with fairness of sync queues áwhere queues are not continuously backlogged. Looks like idling is áonly solution to make sure such kind of queues can get some decent amount áof disk bandwidth in the face of competion from continusouly backlogged áqueues. But excessive idling has potential to reduce performance on SSD áand disks with commnad queuing.

Quoted 7 lines
Quoted 3 lines

o This patch experiments with waiting for next request to come before a áqueue is expired after it has consumed its time slice. This can ensure ámore accurate fairness numbers in some cases.

Vivek, have you introduced this option just to play with it, or you are planning to make it a part of the patch set. Waiting for a new request to come before expiring time slice sounds problematic.

Why are the issues you forsee with it. This is just an extra 8ms idling on the sync queue that is also if think time of the queue is not high.

We already do idling on sync queues. In this case we are doing an extra idle even if queue has consumed its allocated quota. It helps me get fairness numbers and I have put it under a tunable "fairness". So by default this code will not kick in.

Quoted 5 lines

Other possible option could be that when expiring a sync queue, don't remove the queue immediately from the tree and remove it later if there is no request from the queue in 8ms or so. I am not sure with BFQ, is it feasible to do that without creating issues with current implementation. Current implementation was simple, so I stick to it to begin with.

If the maximum wait is bounded by 8ms, then it should be fine. The comments on the patch did not talk about such limit; it sounded like unbounded wait to me.

Does keeping the sync queue in ready tree solves the problem too? Is it because it avoid a virtual time jump?

Quoted 245 lines

So yes, I am planning to keep it under tunable, unless there are significant issues in doing that.

Thanks
Vivek

Quoted 237 lines
Quoted 237 lines

o Introduced a tunable "fairness". If set, io-controller will put more áfocus on getting fairness right than getting throughput right.

---
áblock/blk-sysfs.c á | á á7 ++++
áblock/elevator-fq.c | á 85 +++++++++++++++++++++++++++++++++++++++++++++------- áblock/elevator-fq.h | á 12 +++++++
á3 files changed, 94 insertions(+), 10 deletions(-)

Index: linux1/block/elevator-fq.h
===================================================================

block/elevator-fq.h á2009-03-18 17:34:53.000000000 -0400 renamed from block/elevator-fq.h á á 2009-03-18 17:34:46.000000000 -0400 +0 -0
@@ -318,6 +318,13 @@ struct elv_fq_data {

á á á áunsigned long long rate_sampling_start; /*sampling window start jifies*/ á á á á/* number of sectors finished io during current sampling window */ á á á áunsigned long rate_sectors_current;
+
+ á á á /*
+ á á á á* If set to 1, will disable many optimizations done for boost
+ á á á á* throughput and focus more on providing fairness for sync
+ á á á á* queues.
+ á á á á*/
+ á á á int fairness;
á};

áextern int elv_slice_idle;

block/elevator-fq.h á2009-03-18 17:34:53.000000000 -0400 renamed from block/elevator-fq.h á á 2009-03-18 17:34:46.000000000 -0400 +0 -0
@@ -340,6 +347,7 @@ enum elv_queue_state_flags {

á á á áELV_QUEUE_FLAG_idle_window, á á á /* elevator slice idling enabled */ á á á áELV_QUEUE_FLAG_wait_request, á á á/* waiting for a request */ á á á áELV_QUEUE_FLAG_slice_new, á á á á /* no requests dispatched in slice */
+ á á á ELV_QUEUE_FLAG_wait_busy, á á á á /* wait for this queue to get busy */ á á á áELV_QUEUE_FLAG_NR,
á};

block/elevator-fq.h á2009-03-18 17:34:53.000000000 -0400 renamed from block/elevator-fq.h á á 2009-03-18 17:34:46.000000000 -0400 +1 -0
@@ -362,6 +370,7 @@ ELV_IO_QUEUE_FLAG_FNS(sync)
 áELV_IO_QUEUE_FLAG_FNS(wait_request)
 áELV_IO_QUEUE_FLAG_FNS(idle_window)
 áELV_IO_QUEUE_FLAG_FNS(slice_new)
+ELV_IO_QUEUE_FLAG_FNS(wait_busy)
 

ástatic inline struct io_service_tree *
áio_entity_service_tree(struct io_entity *entity)

block/elevator-fq.h á2009-03-18 17:34:53.000000000 -0400 renamed from block/elevator-fq.h á á 2009-03-18 17:34:46.000000000 -0400 +3 -0
@@ -554,6 +563,9 @@ static inline struct io_queue *elv_looku
 áextern ssize_t elv_slice_idle_show(struct request_queue *q, char *name);
 áextern ssize_t elv_slice_idle_store(struct request_queue *q, const char *name,
 á á á á á á á á á á á á á á á á á á á á á á á ásize_t count);
+extern ssize_t elv_fairness_show(struct request_queue *q, char *name);
+extern ssize_t elv_fairness_store(struct request_queue *q, const char *name,
+ á á á á á á á á á á á á á á á á á á á á á á á size_t count);
 

á/* Functions used by elevator.c */
áextern int elv_init_fq_data(struct request_queue *q, struct elevator_queue *e);
Index: linux1/block/elevator-fq.c
===================================================================

block/elevator-fq.c á2009-03-18 17:34:53.000000000 -0400 renamed from block/elevator-fq.c á á 2009-03-18 17:34:46.000000000 -0400 +38 -0
@@ -1837,6 +1837,44 @@ void elv_ioq_served(struct io_queue *ioq
 á á á á á á á á á á á áioq->total_service);
 á}
 
+/* Functions to show and store fairness value through sysfs */
+ssize_t elv_fairness_show(struct request_queue *q, char *name)
+{
+ á á á struct elv_fq_data *efqd;
+ á á á unsigned int data;
+ á á á unsigned long flags;
+
+ á á á spin_lock_irqsave(q->queue_lock, flags);
+ á á á efqd = &q->elevator->efqd;
+ á á á data = efqd->fairness;
+ á á á spin_unlock_irqrestore(q->queue_lock, flags);
+ á á á return sprintf(name, "%d\n", data);
+}
+
+ssize_t elv_fairness_store(struct request_queue *q, const char *name,
+ á á á á á á á á á á á á size_t count)
+{
+ á á á struct elv_fq_data *efqd;
+ á á á unsigned int data;
+ á á á unsigned long flags;
+
+ á á á char *p = (char *)name;
+
+ á á á data = simple_strtoul(p, &p, 10);
+
+ á á á if (data < 0)
+ á á á á á á á data = 0;
+ á á á else if (data > INT_MAX)
+ á á á á á á á data = INT_MAX;
+
+ á á á spin_lock_irqsave(q->queue_lock, flags);
+ á á á efqd = &q->elevator->efqd;
+ á á á efqd->fairness = data;
+ á á á spin_unlock_irqrestore(q->queue_lock, flags);
+
+ á á á return count;
+}
+

á/* Functions to show and store elv_idle_slice value through sysfs */ ássize_t elv_slice_idle_show(struct request_queue *q, char *name) á{

block/elevator-fq.c á2009-03-18 17:34:53.000000000 -0400 renamed from block/elevator-fq.c á á 2009-03-18 17:34:46.000000000 -0400 +2 -1
@@ -2263,10 +2301,11 @@ void __elv_ioq_slice_expired(struct requ
 á á á áassert_spin_locked(q->queue_lock);
 á á á áelv_log_ioq(efqd, ioq, "slice expired upd=%d", budget_update);
 
- á á á if (elv_ioq_wait_request(ioq))
+ á á á if (elv_ioq_wait_request(ioq) || elv_ioq_wait_busy(ioq))
 á á á á á á á ádel_timer(&efqd->idle_slice_timer);
 
 á á á áelv_clear_ioq_wait_request(ioq);
+ á á á elv_clear_ioq_wait_busy(ioq);
 

á á á á/*
á á á á * if ioq->slice_end = 0, that means a queue was expired before first

block/elevator-fq.c á2009-03-18 17:34:53.000000000 -0400 renamed from block/elevator-fq.c á á 2009-03-18 17:34:46.000000000 -0400 +0 -0
@@ -2482,8 +2521,9 @@ void elv_ioq_request_add(struct request_

á á á á á á á á * immediately and flag that we must not expire this queue á á á á á á á á * just now
á á á á á á á á */
- á á á á á á á if (elv_ioq_wait_request(ioq)) {
+ á á á á á á á if (elv_ioq_wait_request(ioq) || elv_ioq_wait_busy(ioq)) { á á á á á á á á á á á ádel_timer(&efqd->idle_slice_timer);
+ á á á á á á á á á á á elv_clear_ioq_wait_busy(ioq);
á á á á á á á á á á á áblk_start_queueing(q);
á á á á á á á á}
á á á á} else if (elv_should_preempt(q, ioq, rq)) {

block/elevator-fq.c á2009-03-18 17:34:53.000000000 -0400 renamed from block/elevator-fq.c á á 2009-03-18 17:34:46.000000000 -0400 +3 -0
@@ -2519,6 +2559,9 @@ void elv_idle_slice_timer(unsigned long
 
 á á á áif (ioq) {
 
+ á á á á á á á if (elv_ioq_wait_busy(ioq))
+ á á á á á á á á á á á goto expire;
+

á á á á á á á á/*
á á á á á á á á * expired
á á á á á á á á */

block/elevator-fq.c á2009-03-18 17:34:53.000000000 -0400 renamed from block/elevator-fq.c á á 2009-03-18 17:34:46.000000000 -0400 +1 -1
@@ -2546,7 +2589,7 @@ out_cont:
 á á á áspin_unlock_irqrestore(q->queue_lock, flags);
 á}
 
-void elv_ioq_arm_slice_timer(struct request_queue *q)
+void elv_ioq_arm_slice_timer(struct request_queue *q, int wait_for_busy)
 á{
 á á á ástruct elv_fq_data *efqd = &q->elevator->efqd;
 á á á ástruct io_queue *ioq = elv_active_ioq(q->elevator);
@@ -2563,15 +2606,27 @@ void elv_ioq_arm_slice_timer(struct requ
 á á á á á á á áreturn;
 

á á á á/*
- á á á á* still requests with the driver, don't idle
+ á á á á* idle is disabled, either manually or by past process history á á á á */
- á á á if (efqd->rq_in_driver)
+ á á á if (!efqd->elv_slice_idle || !elv_ioq_idle_window(ioq)) á á á á á á á áreturn;

á á á á/*
- á á á á* idle is disabled, either manually or by past process history
+ á á á á* This queue has consumed its time slice. We are waiting only for
+ á á á á* it to become busy before we select next queue for dispatch. á á á á */
- á á á if (!efqd->elv_slice_idle || !elv_ioq_idle_window(ioq))
+ á á á if (efqd->fairness && wait_for_busy) {
+ á á á á á á á elv_mark_ioq_wait_busy(ioq);
+ á á á á á á á sl = efqd->elv_slice_idle;
+ á á á á á á á mod_timer(&efqd->idle_slice_timer, jiffies + sl);
+ á á á á á á á elv_log(efqd, "arm idle: %lu wait busy=1", sl);
+ á á á á á á á return;
+ á á á }
+
+ á á á /*
+ á á á á* still requests with the driver, don't idle
+ á á á á*/
+ á á á if (efqd->rq_in_driver)
á á á á á á á áreturn;

á á á á/*

block/elevator-fq.c á2009-03-18 17:34:53.000000000 -0400 renamed from block/elevator-fq.c á á 2009-03-18 17:34:46.000000000 -0400 +6 -0
@@ -2628,6 +2683,12 @@ void *elv_fq_select_ioq(struct request_q
 á á á á á á á á}
 á á á á}
 
+ á á á /* We are waiting for this queue to become busy before it expires.*/
+ á á á if (efqd->fairness && elv_ioq_wait_busy(ioq)) {
+ á á á á á á á ioq = NULL;
+ á á á á á á á goto keep_queue;
+ á á á }
+

á á á á/*
á á á á * The active queue has run out of time, expire it and select new. á á á á */

block/elevator-fq.c á2009-03-18 17:34:53.000000000 -0400 renamed from block/elevator-fq.c á á 2009-03-18 17:34:46.000000000 -0400 +7 -3
@@ -2802,10 +2863,14 @@ void elv_ioq_completed_request(struct re
 á á á á á á á á á á á áelv_ioq_set_prio_slice(q, ioq);
 á á á á á á á á á á á áelv_clear_ioq_slice_new(ioq);
 á á á á á á á á}
- á á á á á á á if (elv_ioq_slice_used(ioq) || elv_ioq_class_idle(ioq))
+ á á á á á á á if (elv_ioq_class_idle(ioq))
 á á á á á á á á á á á áelv_ioq_slice_expired(q, 1);
- á á á á á á á else if (sync && !ioq->nr_queued)
- á á á á á á á á á á á elv_ioq_arm_slice_timer(q);
+ á á á á á á á else if (sync && !ioq->nr_queued) {
+ á á á á á á á á á á á if (elv_ioq_slice_used(ioq))
+ á á á á á á á á á á á á á á á elv_ioq_arm_slice_timer(q, 1);
+ á á á á á á á á á á á else
+ á á á á á á á á á á á á á á á elv_ioq_arm_slice_timer(q, 0);
+ á á á á á á á }
 á á á á}
 
 á á á áif (!efqd->rq_in_driver)

Index: linux1/block/blk-sysfs.c
===================================================================

block/blk-sysfs.c á á2009-03-18 17:34:53.000000000 -0400 renamed from block/blk-sysfs.c á á á 2009-03-18 17:34:28.000000000 -0400 +6 -0
@@ -282,6 +282,12 @@ static struct queue_sysfs_entry queue_sl
 á á á á.show = elv_slice_idle_show,
 á á á á.store = elv_slice_idle_store,
 á};
+
+static struct queue_sysfs_entry queue_fairness_entry = {
+ á á á .attr = {.name = "fairness", .mode = S_IRUGO | S_IWUSR },
+ á á á .show = elv_fairness_show,
+ á á á .store = elv_fairness_store,
+};

á#endif
ástatic struct attribute *default_attrs[] = {
á á á á&queue_requests_entry.attr,

block/blk-sysfs.c á á2009-03-18 17:34:53.000000000 -0400 renamed from block/blk-sysfs.c á á á 2009-03-18 17:34:28.000000000 -0400 +0 -0
@@ -296,6 +302,7 @@ static struct attribute *default_attrs[]
 á á á á&queue_iostats_entry.attr,

á#ifdef CONFIG_ELV_FAIR_QUEUING
á á á á&queue_slice_idle_entry.attr,
+ á á á &queue_fairness_entry.attr,
á#endif
á á á áNULL,
á};

#82 lore raw

Re: [PATCH 01/10] Documentation

In reply to Vivek Goyal
Quoted 36 lines · Vivek Goyal

On Mon, Mar 23, 2009 at 10:32:41PM -0700, Nauman Rafique wrote:

[..]

Quoted 17 lines
Quoted 17 lines

DESC
io-controller: idle for sometime on sync queue before expiring it EDESC

o When a sync queue expires, in many cases it might be empty and then áit will be deleted from the active tree. This will lead to a scenario áwhere out of two competing queues, only one is on the tree and when a ánew queue is selected, vtime jump takes place and we don't see services áprovided in proportion to weight.

o In general this is a fundamental problem with fairness of sync queues áwhere queues are not continuously backlogged. Looks like idling is áonly solution to make sure such kind of queues can get some decent amount áof disk bandwidth in the face of competion from continusouly backlogged áqueues. But excessive idling has potential to reduce performance on SSD áand disks with commnad queuing.

Quoted 7 lines
Quoted 3 lines

o This patch experiments with waiting for next request to come before a áqueue is expired after it has consumed its time slice. This can ensure ámore accurate fairness numbers in some cases.

Vivek, have you introduced this option just to play with it, or you are planning to make it a part of the patch set. Waiting for a new request to come before expiring time slice sounds problematic.

Why are the issues you forsee with it. This is just an extra 8ms idling on the sync queue that is also if think time of the queue is not high.

We already do idling on sync queues. In this case we are doing an extra idle even if queue has consumed its allocated quota. It helps me get fairness numbers and I have put it under a tunable "fairness". So by default this code will not kick in.

Quoted 5 lines

Other possible option could be that when expiring a sync queue, don't remove the queue immediately from the tree and remove it later if there is no request from the queue in 8ms or so. I am not sure with BFQ, is it feasible to do that without creating issues with current implementation. Current implementation was simple, so I stick to it to begin with.

If the maximum wait is bounded by 8ms, then it should be fine. The comments on the patch did not talk about such limit; it sounded like unbounded wait to me.

Does keeping the sync queue in ready tree solves the problem too? Is it because it avoid a virtual time jump?

Quoted 245 lines

So yes, I am planning to keep it under tunable, unless there are significant issues in doing that.

Thanks
Vivek

Quoted 237 lines
Quoted 237 lines

o Introduced a tunable "fairness". If set, io-controller will put more áfocus on getting fairness right than getting throughput right.

---
áblock/blk-sysfs.c á | á á7 ++++
áblock/elevator-fq.c | á 85 +++++++++++++++++++++++++++++++++++++++++++++------- áblock/elevator-fq.h | á 12 +++++++
á3 files changed, 94 insertions(+), 10 deletions(-)

Index: linux1/block/elevator-fq.h
===================================================================

block/elevator-fq.h á2009-03-18 17:34:53.000000000 -0400 renamed from block/elevator-fq.h á á 2009-03-18 17:34:46.000000000 -0400 +0 -0
@@ -318,6 +318,13 @@ struct elv_fq_data {

á á á áunsigned long long rate_sampling_start; /*sampling window start jifies*/ á á á á/* number of sectors finished io during current sampling window */ á á á áunsigned long rate_sectors_current;
+
+ á á á /*
+ á á á á* If set to 1, will disable many optimizations done for boost
+ á á á á* throughput and focus more on providing fairness for sync
+ á á á á* queues.
+ á á á á*/
+ á á á int fairness;
á};

áextern int elv_slice_idle;

block/elevator-fq.h á2009-03-18 17:34:53.000000000 -0400 renamed from block/elevator-fq.h á á 2009-03-18 17:34:46.000000000 -0400 +0 -0
@@ -340,6 +347,7 @@ enum elv_queue_state_flags {

á á á áELV_QUEUE_FLAG_idle_window, á á á /* elevator slice idling enabled */ á á á áELV_QUEUE_FLAG_wait_request, á á á/* waiting for a request */ á á á áELV_QUEUE_FLAG_slice_new, á á á á /* no requests dispatched in slice */
+ á á á ELV_QUEUE_FLAG_wait_busy, á á á á /* wait for this queue to get busy */ á á á áELV_QUEUE_FLAG_NR,
á};

block/elevator-fq.h á2009-03-18 17:34:53.000000000 -0400 renamed from block/elevator-fq.h á á 2009-03-18 17:34:46.000000000 -0400 +1 -0
@@ -362,6 +370,7 @@ ELV_IO_QUEUE_FLAG_FNS(sync)
 áELV_IO_QUEUE_FLAG_FNS(wait_request)
 áELV_IO_QUEUE_FLAG_FNS(idle_window)
 áELV_IO_QUEUE_FLAG_FNS(slice_new)
+ELV_IO_QUEUE_FLAG_FNS(wait_busy)
 

ástatic inline struct io_service_tree *
áio_entity_service_tree(struct io_entity *entity)

block/elevator-fq.h á2009-03-18 17:34:53.000000000 -0400 renamed from block/elevator-fq.h á á 2009-03-18 17:34:46.000000000 -0400 +3 -0
@@ -554,6 +563,9 @@ static inline struct io_queue *elv_looku
 áextern ssize_t elv_slice_idle_show(struct request_queue *q, char *name);
 áextern ssize_t elv_slice_idle_store(struct request_queue *q, const char *name,
 á á á á á á á á á á á á á á á á á á á á á á á ásize_t count);
+extern ssize_t elv_fairness_show(struct request_queue *q, char *name);
+extern ssize_t elv_fairness_store(struct request_queue *q, const char *name,
+ á á á á á á á á á á á á á á á á á á á á á á á size_t count);
 

á/* Functions used by elevator.c */
áextern int elv_init_fq_data(struct request_queue *q, struct elevator_queue *e);
Index: linux1/block/elevator-fq.c
===================================================================

block/elevator-fq.c á2009-03-18 17:34:53.000000000 -0400 renamed from block/elevator-fq.c á á 2009-03-18 17:34:46.000000000 -0400 +38 -0
@@ -1837,6 +1837,44 @@ void elv_ioq_served(struct io_queue *ioq
 á á á á á á á á á á á áioq->total_service);
 á}
 
+/* Functions to show and store fairness value through sysfs */
+ssize_t elv_fairness_show(struct request_queue *q, char *name)
+{
+ á á á struct elv_fq_data *efqd;
+ á á á unsigned int data;
+ á á á unsigned long flags;
+
+ á á á spin_lock_irqsave(q->queue_lock, flags);
+ á á á efqd = &q->elevator->efqd;
+ á á á data = efqd->fairness;
+ á á á spin_unlock_irqrestore(q->queue_lock, flags);
+ á á á return sprintf(name, "%d\n", data);
+}
+
+ssize_t elv_fairness_store(struct request_queue *q, const char *name,
+ á á á á á á á á á á á á size_t count)
+{
+ á á á struct elv_fq_data *efqd;
+ á á á unsigned int data;
+ á á á unsigned long flags;
+
+ á á á char *p = (char *)name;
+
+ á á á data = simple_strtoul(p, &p, 10);
+
+ á á á if (data < 0)
+ á á á á á á á data = 0;
+ á á á else if (data > INT_MAX)
+ á á á á á á á data = INT_MAX;
+
+ á á á spin_lock_irqsave(q->queue_lock, flags);
+ á á á efqd = &q->elevator->efqd;
+ á á á efqd->fairness = data;
+ á á á spin_unlock_irqrestore(q->queue_lock, flags);
+
+ á á á return count;
+}
+

á/* Functions to show and store elv_idle_slice value through sysfs */ ássize_t elv_slice_idle_show(struct request_queue *q, char *name) á{

block/elevator-fq.c á2009-03-18 17:34:53.000000000 -0400 renamed from block/elevator-fq.c á á 2009-03-18 17:34:46.000000000 -0400 +2 -1
@@ -2263,10 +2301,11 @@ void __elv_ioq_slice_expired(struct requ
 á á á áassert_spin_locked(q->queue_lock);
 á á á áelv_log_ioq(efqd, ioq, "slice expired upd=%d", budget_update);
 
- á á á if (elv_ioq_wait_request(ioq))
+ á á á if (elv_ioq_wait_request(ioq) || elv_ioq_wait_busy(ioq))
 á á á á á á á ádel_timer(&efqd->idle_slice_timer);
 
 á á á áelv_clear_ioq_wait_request(ioq);
+ á á á elv_clear_ioq_wait_busy(ioq);
 

á á á á/*
á á á á * if ioq->slice_end = 0, that means a queue was expired before first

block/elevator-fq.c á2009-03-18 17:34:53.000000000 -0400 renamed from block/elevator-fq.c á á 2009-03-18 17:34:46.000000000 -0400 +0 -0
@@ -2482,8 +2521,9 @@ void elv_ioq_request_add(struct request_

á á á á á á á á * immediately and flag that we must not expire this queue á á á á á á á á * just now
á á á á á á á á */
- á á á á á á á if (elv_ioq_wait_request(ioq)) {
+ á á á á á á á if (elv_ioq_wait_request(ioq) || elv_ioq_wait_busy(ioq)) { á á á á á á á á á á á ádel_timer(&efqd->idle_slice_timer);
+ á á á á á á á á á á á elv_clear_ioq_wait_busy(ioq);
á á á á á á á á á á á áblk_start_queueing(q);
á á á á á á á á}
á á á á} else if (elv_should_preempt(q, ioq, rq)) {

block/elevator-fq.c á2009-03-18 17:34:53.000000000 -0400 renamed from block/elevator-fq.c á á 2009-03-18 17:34:46.000000000 -0400 +3 -0
@@ -2519,6 +2559,9 @@ void elv_idle_slice_timer(unsigned long
 
 á á á áif (ioq) {
 
+ á á á á á á á if (elv_ioq_wait_busy(ioq))
+ á á á á á á á á á á á goto expire;
+

á á á á á á á á/*
á á á á á á á á * expired
á á á á á á á á */

block/elevator-fq.c á2009-03-18 17:34:53.000000000 -0400 renamed from block/elevator-fq.c á á 2009-03-18 17:34:46.000000000 -0400 +1 -1
@@ -2546,7 +2589,7 @@ out_cont:
 á á á áspin_unlock_irqrestore(q->queue_lock, flags);
 á}
 
-void elv_ioq_arm_slice_timer(struct request_queue *q)
+void elv_ioq_arm_slice_timer(struct request_queue *q, int wait_for_busy)
 á{
 á á á ástruct elv_fq_data *efqd = &q->elevator->efqd;
 á á á ástruct io_queue *ioq = elv_active_ioq(q->elevator);
@@ -2563,15 +2606,27 @@ void elv_ioq_arm_slice_timer(struct requ
 á á á á á á á áreturn;
 

á á á á/*
- á á á á* still requests with the driver, don't idle
+ á á á á* idle is disabled, either manually or by past process history á á á á */
- á á á if (efqd->rq_in_driver)
+ á á á if (!efqd->elv_slice_idle || !elv_ioq_idle_window(ioq)) á á á á á á á áreturn;

á á á á/*
- á á á á* idle is disabled, either manually or by past process history
+ á á á á* This queue has consumed its time slice. We are waiting only for
+ á á á á* it to become busy before we select next queue for dispatch. á á á á */
- á á á if (!efqd->elv_slice_idle || !elv_ioq_idle_window(ioq))
+ á á á if (efqd->fairness && wait_for_busy) {
+ á á á á á á á elv_mark_ioq_wait_busy(ioq);
+ á á á á á á á sl = efqd->elv_slice_idle;
+ á á á á á á á mod_timer(&efqd->idle_slice_timer, jiffies + sl);
+ á á á á á á á elv_log(efqd, "arm idle: %lu wait busy=1", sl);
+ á á á á á á á return;
+ á á á }
+
+ á á á /*
+ á á á á* still requests with the driver, don't idle
+ á á á á*/
+ á á á if (efqd->rq_in_driver)
á á á á á á á áreturn;

á á á á/*

block/elevator-fq.c á2009-03-18 17:34:53.000000000 -0400 renamed from block/elevator-fq.c á á 2009-03-18 17:34:46.000000000 -0400 +6 -0
@@ -2628,6 +2683,12 @@ void *elv_fq_select_ioq(struct request_q
 á á á á á á á á}
 á á á á}
 
+ á á á /* We are waiting for this queue to become busy before it expires.*/
+ á á á if (efqd->fairness && elv_ioq_wait_busy(ioq)) {
+ á á á á á á á ioq = NULL;
+ á á á á á á á goto keep_queue;
+ á á á }
+

á á á á/*
á á á á * The active queue has run out of time, expire it and select new. á á á á */

block/elevator-fq.c á2009-03-18 17:34:53.000000000 -0400 renamed from block/elevator-fq.c á á 2009-03-18 17:34:46.000000000 -0400 +7 -3
@@ -2802,10 +2863,14 @@ void elv_ioq_completed_request(struct re
 á á á á á á á á á á á áelv_ioq_set_prio_slice(q, ioq);
 á á á á á á á á á á á áelv_clear_ioq_slice_new(ioq);
 á á á á á á á á}
- á á á á á á á if (elv_ioq_slice_used(ioq) || elv_ioq_class_idle(ioq))
+ á á á á á á á if (elv_ioq_class_idle(ioq))
 á á á á á á á á á á á áelv_ioq_slice_expired(q, 1);
- á á á á á á á else if (sync && !ioq->nr_queued)
- á á á á á á á á á á á elv_ioq_arm_slice_timer(q);
+ á á á á á á á else if (sync && !ioq->nr_queued) {
+ á á á á á á á á á á á if (elv_ioq_slice_used(ioq))
+ á á á á á á á á á á á á á á á elv_ioq_arm_slice_timer(q, 1);
+ á á á á á á á á á á á else
+ á á á á á á á á á á á á á á á elv_ioq_arm_slice_timer(q, 0);
+ á á á á á á á }
 á á á á}
 
 á á á áif (!efqd->rq_in_driver)

Index: linux1/block/blk-sysfs.c
===================================================================

block/blk-sysfs.c á á2009-03-18 17:34:53.000000000 -0400 renamed from block/blk-sysfs.c á á á 2009-03-18 17:34:28.000000000 -0400 +6 -0
@@ -282,6 +282,12 @@ static struct queue_sysfs_entry queue_sl
 á á á á.show = elv_slice_idle_show,
 á á á á.store = elv_slice_idle_store,
 á};
+
+static struct queue_sysfs_entry queue_fairness_entry = {
+ á á á .attr = {.name = "fairness", .mode = S_IRUGO | S_IWUSR },
+ á á á .show = elv_fairness_show,
+ á á á .store = elv_fairness_store,
+};

á#endif
ástatic struct attribute *default_attrs[] = {
á á á á&queue_requests_entry.attr,

block/blk-sysfs.c á á2009-03-18 17:34:53.000000000 -0400 renamed from block/blk-sysfs.c á á á 2009-03-18 17:34:28.000000000 -0400 +0 -0
@@ -296,6 +302,7 @@ static struct attribute *default_attrs[]
 á á á á&queue_iostats_entry.attr,

á#ifdef CONFIG_ELV_FAIR_QUEUING
á á á á&queue_slice_idle_entry.attr,
+ á á á &queue_fairness_entry.attr,
á#endif
á á á áNULL,
á};

#83 lore raw

Re: [PATCH 01/10] Documentation

Quoted 43 lines · Nauman Rafique
Quoted 36 lines · Vivek Goyal

On Mon, Mar 23, 2009 at 10:32:41PM -0700, Nauman Rafique wrote:

[..]

Quoted 17 lines
Quoted 17 lines

DESC
io-controller: idle for sometime on sync queue before expiring it EDESC

o When a sync queue expires, in many cases it might be empty and then áit will be deleted from the active tree. This will lead to a scenario áwhere out of two competing queues, only one is on the tree and when a ánew queue is selected, vtime jump takes place and we don't see services áprovided in proportion to weight.

o In general this is a fundamental problem with fairness of sync queues áwhere queues are not continuously backlogged. Looks like idling is áonly solution to make sure such kind of queues can get some decent amount áof disk bandwidth in the face of competion from continusouly backlogged áqueues. But excessive idling has potential to reduce performance on SSD áand disks with commnad queuing.

Quoted 7 lines
Quoted 3 lines

o This patch experiments with waiting for next request to come before a áqueue is expired after it has consumed its time slice. This can ensure ámore accurate fairness numbers in some cases.

Vivek, have you introduced this option just to play with it, or you are planning to make it a part of the patch set. Waiting for a new request to come before expiring time slice sounds problematic.

Why are the issues you forsee with it. This is just an extra 8ms idling on the sync queue that is also if think time of the queue is not high.

We already do idling on sync queues. In this case we are doing an extra idle even if queue has consumed its allocated quota. It helps me get fairness numbers and I have put it under a tunable "fairness". So by default this code will not kick in.

Quoted 5 lines

Other possible option could be that when expiring a sync queue, don't remove the queue immediately from the tree and remove it later if there is no request from the queue in 8ms or so. I am not sure with BFQ, is it feasible to do that without creating issues with current implementation. Current implementation was simple, so I stick to it to begin with.

Quoted 7 lines

If the maximum wait is bounded by 8ms, then it should be fine. The comments on the patch did not talk about such limit; it sounded like unbounded wait to me.

Does keeping the sync queue in ready tree solves the problem too? Is it because it avoid a virtual time jump?

I have not tried the second approch yet. But that also should solve the vtime jump issue.

Thanks
Vivek

#84 lore raw

Re: [PATCH 01/10] Documentation

Quoted 3 lines

From: Vivek Goyal <[email protected]>
Date: Tue, Mar 24, 2009 02:29:06PM -0400

...

Quoted 7 lines
Quoted 3 lines

Does keeping the sync queue in ready tree solves the problem too? Is it because it avoid a virtual time jump?

I have not tried the second approch yet. But that also should solve the vtime jump issue.

Do you mean that you intend to keep a queue with no backlog in the active tree?

#85 lore raw

Re: [PATCH 01/10] Documentation

Quoted 8 lines · Fabio Checconi
Quoted 3 lines

From: Vivek Goyal <[email protected]>
Date: Tue, Mar 24, 2009 02:29:06PM -0400

...

Quoted 4 lines
Quoted 3 lines

Does keeping the sync queue in ready tree solves the problem too? Is it because it avoid a virtual time jump?

Quoted 6 lines
Quoted 3 lines

I have not tried the second approch yet. But that also should solve the vtime jump issue.

Do you mean that you intend to keep a queue with no backlog in the active tree?

Yes. Is it possible to keep a not-backlogged queue in the tree for later expiry. So that we don't actively wait/idle for next request to come and hope queue will become backlogged soon. Otherwise, it will be deleted from the active queue. This is just a thought, I am not even sure how would it interefere with bfq code.

All this to solve the vtime jump issue for sync queues.

Thanks
Vivek

A message that is not in the archive (sent off-list or not yet received): [email protected]

#87 lore raw

Re: [PATCH 01/10] Documentation

Quoted 16 lines · Vivek Goyal
Quoted 8 lines · Fabio Checconi
Quoted 3 lines

From: Vivek Goyal <[email protected]>
Date: Tue, Mar 24, 2009 02:29:06PM -0400

...

Quoted 4 lines
Quoted 3 lines

Does keeping the sync queue in ready tree solves the problem too? Is it because it avoid a virtual time jump?

Quoted 6 lines
Quoted 3 lines

I have not tried the second approch yet. But that also should solve the vtime jump issue.

Do you mean that you intend to keep a queue with no backlog in the active tree?

Quoted 7 lines

Yes. Is it possible to keep a not-backlogged queue in the tree for later expiry. So that we don't actively wait/idle for next request to come and hope queue will become backlogged soon. Otherwise, it will be deleted from the active queue. This is just a thought, I am not even sure how would it interefere with bfq code.

All this to solve the vtime jump issue for sync queues.

If only vtime jump is an issue, can we solve it by delaying vtime jump? That is, even if we serve an entity with a bigger vtime, we don't update the reference vtime of the service tree until after some time?

Quoted 4 lines

Thanks
Vivek

#88 lore raw

Re: [PATCH 01/10] Documentation

In reply to a message not in the archive
Quoted 19 lines

From: Vivek Goyal <[email protected]>
Date: Tue, Mar 24, 2009 02:35:32PM -0400

Quoted 8 lines · Fabio Checconi
Quoted 3 lines

From: Vivek Goyal <[email protected]>
Date: Tue, Mar 24, 2009 02:29:06PM -0400

...

Quoted 4 lines
Quoted 3 lines

Does keeping the sync queue in ready tree solves the problem too? Is it because it avoid a virtual time jump?

Quoted 6 lines
Quoted 3 lines

I have not tried the second approch yet. But that also should solve the vtime jump issue.

Do you mean that you intend to keep a queue with no backlog in the active tree?

Quoted 8 lines

Yes. Is it possible to keep a not-backlogged queue in the tree for later expiry. So that we don't actively wait/idle for next request to come and hope queue will become backlogged soon. Otherwise, it will be deleted from the active queue. This is just a thought, I am not even sure how would it interefere with bfq code.

All this to solve the vtime jump issue for sync queues.

Of course it is possible, but if you stick with wf2q+ the virtual time will jump anyway, and the gain would be that each scheduling decision will have O(N logN) complexity instead of O(log N), to skip empty queues.

Otherwise, if you'll do your own timestamping (where any new request can get a timestamp smaller that the virtual time) then nothing from the theory BFQ was based on can give any hint on the guarantees that the resulting algorithm can provide.

#89 lore raw

Re: [PATCH 01/10] Documentation

In reply to Vivek Goyal
Quoted 16 lines · Vivek Goyal
Quoted 8 lines · Fabio Checconi
Quoted 3 lines

From: Vivek Goyal <[email protected]>
Date: Tue, Mar 24, 2009 02:29:06PM -0400

...

Quoted 4 lines
Quoted 3 lines

Does keeping the sync queue in ready tree solves the problem too? Is it because it avoid a virtual time jump?

Quoted 6 lines
Quoted 3 lines

I have not tried the second approch yet. But that also should solve the vtime jump issue.

Do you mean that you intend to keep a queue with no backlog in the active tree?

Quoted 7 lines

Yes. Is it possible to keep a not-backlogged queue in the tree for later expiry. So that we don't actively wait/idle for next request to come and hope queue will become backlogged soon. Otherwise, it will be deleted from the active queue. This is just a thought, I am not even sure how would it interefere with bfq code.

All this to solve the vtime jump issue for sync queues.

If only vtime jump is an issue, can we solve it by delaying vtime jump? That is, even if we serve an entity with a bigger vtime, we don't update the reference vtime of the service tree until after some time?

Quoted 4 lines

Thanks
Vivek

#90 lore raw

Re: [PATCH 01/10] Documentation

In reply to Vivek Goyal
Quoted 19 lines

From: Vivek Goyal <[email protected]>
Date: Tue, Mar 24, 2009 02:35:32PM -0400

Quoted 8 lines · Fabio Checconi
Quoted 3 lines

From: Vivek Goyal <[email protected]>
Date: Tue, Mar 24, 2009 02:29:06PM -0400

...

Quoted 4 lines
Quoted 3 lines

Does keeping the sync queue in ready tree solves the problem too? Is it because it avoid a virtual time jump?

Quoted 6 lines
Quoted 3 lines

I have not tried the second approch yet. But that also should solve the vtime jump issue.

Do you mean that you intend to keep a queue with no backlog in the active tree?

Quoted 8 lines

Yes. Is it possible to keep a not-backlogged queue in the tree for later expiry. So that we don't actively wait/idle for next request to come and hope queue will become backlogged soon. Otherwise, it will be deleted from the active queue. This is just a thought, I am not even sure how would it interefere with bfq code.

All this to solve the vtime jump issue for sync queues.

Of course it is possible, but if you stick with wf2q+ the virtual time will jump anyway, and the gain would be that each scheduling decision will have O(N logN) complexity instead of O(log N), to skip empty queues.

Otherwise, if you'll do your own timestamping (where any new request can get a timestamp smaller that the virtual time) then nothing from the theory BFQ was based on can give any hint on the guarantees that the resulting algorithm can provide.

A message that is not in the archive (sent off-list or not yet received): [email protected]

#92 lore raw

Re: [PATCH 01/10] Documentation

Quoted 3 lines

From: Vivek Goyal <[email protected]>
Date: Tue, Mar 24, 2009 02:29:06PM -0400

...

Quoted 7 lines
Quoted 3 lines

Does keeping the sync queue in ready tree solves the problem too? Is it because it avoid a virtual time jump?

I have not tried the second approch yet. But that also should solve the vtime jump issue.

Do you mean that you intend to keep a queue with no backlog in the active tree?

#93 lore raw

Re: [PATCH 01/10] Documentation

In reply to Vivek Goyal
Quoted 27 lines · Vivek Goyal
Documentation/block/io-controller.txt new file +27 -0
+Going back to old behavior
+==========================
+In new scheme of things essentially we are creating hierarchical fair
+queuing logic in elevator layer and changing IO schedulers to make use of
+that logic so that end IO schedulers start supporting hierarchical scheduling.
+
+Elevator layer continues to support the old interfaces. So even if fair queuing
+is enabled at elevator layer, one can have both new hierarchical scheduler as
+well as old non-hierarchical scheduler operating.
+
+Also noop, deadline and AS have option of enabling hierarchical scheduling.
+If it is selected, fair queuing is done in hierarchical manner. If hierarchical
+scheduling is disabled, noop, deadline and AS should retain their existing
+behavior.
+
+CFQ is the only exception where one can not disable fair queuing as it is
+needed for providing fairness among various threads even in non-hierarchical
+mode.
+
+Various user visible config options
+===================================
+CONFIG_IOSCHED_NOOP_HIER
+       - Enables hierchical fair queuing in noop. Not selecting this option
+         leads to old behavior of noop.
+
+CONFIG_IOSCHED_DEADLINE_HIER
+       - Enables hierchical fair queuing in deadline. Not selecting this

Documentation/block/io-controller.txt line 152

Quoted 10 lines
Documentation/block/io-controller.txt new file +10 -0
+         option leads to old behavior of deadline.
+
+CONFIG_IOSCHED_AS_HIER
+       - Enables hierchical fair queuing in AS. Not selecting this option
+         leads to old behavior of AS.
+
+CONFIG_IOSCHED_CFQ_HIER
+       - Enables hierarchical fair queuing in CFQ. Not selecting this option
+         still does fair queuing among various queus but it is flat and not
+         hierarchical.

One worry I have is that these are compile time switches. Is there any way you can get the old AS/DEADLINE back when these are enabled but you're not actively using cgroups?

#94 lore raw

Re: [PATCH 01/10] Documentation

Quoted 34 lines · Peter Zijlstra
Quoted 33 lines · Vivek Goyal
Documentation/block/io-controller.txt new file +33 -0
+Going back to old behavior
+==========================
+In new scheme of things essentially we are creating hierarchical fair
+queuing logic in elevator layer and changing IO schedulers to make use of
+that logic so that end IO schedulers start supporting hierarchical scheduling.
+
+Elevator layer continues to support the old interfaces. So even if fair queuing
+is enabled at elevator layer, one can have both new hierarchical scheduler as
+well as old non-hierarchical scheduler operating.
+
+Also noop, deadline and AS have option of enabling hierarchical scheduling.
+If it is selected, fair queuing is done in hierarchical manner. If hierarchical
+scheduling is disabled, noop, deadline and AS should retain their existing
+behavior.
+
+CFQ is the only exception where one can not disable fair queuing as it is
+needed for providing fairness among various threads even in non-hierarchical
+mode.
+
+Various user visible config options
+===================================
+CONFIG_IOSCHED_NOOP_HIER
+       - Enables hierchical fair queuing in noop. Not selecting this option
+         leads to old behavior of noop.
+
+CONFIG_IOSCHED_DEADLINE_HIER
+       - Enables hierchical fair queuing in deadline. Not selecting this
+         option leads to old behavior of deadline.
+
+CONFIG_IOSCHED_AS_HIER
+       - Enables hierchical fair queuing in AS. Not selecting this option
+         leads to old behavior of AS.
+
Quoted 8 lines

Documentation/block/io-controller.txt line 152

Quoted 4 lines
Documentation/block/io-controller.txt new file +4 -0
+CONFIG_IOSCHED_CFQ_HIER
+       - Enables hierarchical fair queuing in CFQ. Not selecting this option
+         still does fair queuing among various queus but it is flat and not
+         hierarchical.

One worry I have is that these are compile time switches. Is there any way you can get the old AS/DEADLINE back when these are enabled but you're not actively using cgroups?

Hi Peter,

In principle, if one is not using cgroups, there is only one io queue in the root group and most likely we should achieve the same behavior as old schedulers. Just that some extra code gets into execution at runtime.

I have not got a chance yet to do some numbers but I think this path can be optimized enough that at run time effectively we don't see any significant performance penalty and behavior of schedulers is almost same as old ones.

Thanks
Vivek

#95 lore raw

Re: [PATCH 01/10] Documentation

In reply to Peter Zijlstra
Quoted 34 lines · Peter Zijlstra
Quoted 33 lines · Vivek Goyal
Documentation/block/io-controller.txt new file +33 -0
+Going back to old behavior
+==========================
+In new scheme of things essentially we are creating hierarchical fair
+queuing logic in elevator layer and changing IO schedulers to make use of
+that logic so that end IO schedulers start supporting hierarchical scheduling.
+
+Elevator layer continues to support the old interfaces. So even if fair queuing
+is enabled at elevator layer, one can have both new hierarchical scheduler as
+well as old non-hierarchical scheduler operating.
+
+Also noop, deadline and AS have option of enabling hierarchical scheduling.
+If it is selected, fair queuing is done in hierarchical manner. If hierarchical
+scheduling is disabled, noop, deadline and AS should retain their existing
+behavior.
+
+CFQ is the only exception where one can not disable fair queuing as it is
+needed for providing fairness among various threads even in non-hierarchical
+mode.
+
+Various user visible config options
+===================================
+CONFIG_IOSCHED_NOOP_HIER
+       - Enables hierchical fair queuing in noop. Not selecting this option
+         leads to old behavior of noop.
+
+CONFIG_IOSCHED_DEADLINE_HIER
+       - Enables hierchical fair queuing in deadline. Not selecting this
+         option leads to old behavior of deadline.
+
+CONFIG_IOSCHED_AS_HIER
+       - Enables hierchical fair queuing in AS. Not selecting this option
+         leads to old behavior of AS.
+
Quoted 8 lines

Documentation/block/io-controller.txt line 152

Quoted 4 lines
Documentation/block/io-controller.txt new file +4 -0
+CONFIG_IOSCHED_CFQ_HIER
+       - Enables hierarchical fair queuing in CFQ. Not selecting this option
+         still does fair queuing among various queus but it is flat and not
+         hierarchical.

One worry I have is that these are compile time switches. Is there any way you can get the old AS/DEADLINE back when these are enabled but you're not actively using cgroups?

Hi Peter,

In principle, if one is not using cgroups, there is only one io queue in the root group and most likely we should achieve the same behavior as old schedulers. Just that some extra code gets into execution at runtime.

I have not got a chance yet to do some numbers but I think this path can be optimized enough that at run time effectively we don't see any significant performance penalty and behavior of schedulers is almost same as old ones.

Thanks
Vivek

#96 lore raw

Re: [PATCH 01/10] Documentation

In reply to Vivek Goyal
Quoted 49 lines · Vivek Goyal

o Documentation for io-controller.

---
 Documentation/block/io-controller.txt |  221 +++++++++++++++++++++++++++++++++
 1 files changed, 221 insertions(+), 0 deletions(-)
 create mode 100644 Documentation/block/io-controller.txt
Documentation/block/io-controller.txt new file +35 -0
new file mode 100644
index 0000000..8884c5a
@@ -0,0 +1,221 @@
+				IO Controller
+				=============
+
+Overview
+========
+
+This patchset implements a proportional weight IO controller. That is one
+can create cgroups and assign prio/weights to those cgroups and task group
+will get access to disk proportionate to the weight of the group.
+
+These patches modify elevator layer and individual IO schedulers to do
+IO control hence this io controller works only on block devices which use
+one of the standard io schedulers can not be used with any xyz logical block
+device.
+
+The assumption/thought behind modifying IO scheduler is that resource control
+is needed only on leaf nodes where the actual contention for resources is
+present and not on intertermediate logical block devices.
+
+Consider following hypothetical scenario. Lets say there are three physical
+disks, namely sda, sdb and sdc. Two logical volumes (lv0 and lv1) have been
+created on top of these. Some part of sdb is in lv0 and some part is in lv1.
+
+			    lv0      lv1
+			  /	\  /     \
+			sda      sdb      sdc
+
+Also consider following cgroup hierarchy
+
+				root
+				/   \
+			       A     B
+			      / \    / \
+			     T1 T2  T3  T4
+

Documentation/block/io-controller.txt line 45

Quoted 10 lines
Documentation/block/io-controller.txt new file +10 -0
+A and B are two cgroups and T1, T2, T3 and T4 are tasks with-in those cgroups.
+Assuming T1, T2, T3 and T4 are doing IO on lv0 and lv1. These tasks should
+get their fair share of bandwidth on disks sda, sdb and sdc. There is no
+IO control on intermediate logical block nodes (lv0, lv1).
+
+So if tasks T1 and T2 are doing IO on lv0 and T3 and T4 are doing IO on lv1
+only, there will not be any contetion for resources between group A and B if
+IO is going to sda or sdc. But if actual IO gets translated to disk sdb, then
+IO scheduler associated with the sdb will distribute disk bandwidth to
+group A and B proportionate to their weight.

What if we have partitions sda1, sda2 and sda3 instead of sda, sdb and sdc?

Quoted 9 lines
Documentation/block/io-controller.txt new file +9 -0
+
+CFQ already has the notion of fairness and it provides differential disk
+access based on priority and class of the task. Just that it is flat and
+with cgroup stuff, it needs to be made hierarchical.
+
+Rest of the IO schedulers (noop, deadline and AS) don't have any notion
+of fairness among various threads.
+
+One of the concerns raised with modifying IO schedulers was that we don't

Documentation/block/io-controller.txt line 64

Quoted 10 lines
Documentation/block/io-controller.txt new file +10 -0
+want to replicate the code in all the IO schedulers. These patches share
+the fair queuing code which has been moved to a common layer (elevator
+layer). Hence we don't end up replicating code across IO schedulers.
+
+Design
+======
+This patchset primarily uses BFQ (Budget Fair Queuing) code to provide
+fairness among different IO queues. Fabio and Paolo implemented BFQ which uses
+B-WF2Q+ algorithm for fair queuing.
+

References to BFQ, please. I can search them, but having them in the doc would be nice.

Quoted 4 lines
Documentation/block/io-controller.txt new file +4 -0
+Why BFQ?
+
+- Not sure if weighted round robin logic of CFQ can be easily extended for
+  hierarchical mode. One of the things is that we can not keep dividing

Documentation/block/io-controller.txt line 78

Quoted 10 lines
Documentation/block/io-controller.txt new file +10 -0
+  the time slice of parent group among childrens. Deeper we go in hierarchy
+  time slice will get smaller.
+
+  One of the ways to implement hierarchical support could be to keep track
+  of virtual time and service provided to queue/group and select a queue/group
+  for service based on any of the various available algoriths.
+
+  BFQ already had support for hierarchical scheduling, taking those patches
+  was easier.
+

Could you elaborate, when you say timeslices get smaller -

1. Are you referring to inability to use higher resolution time?
2. Loss of throughput due to timeslice degradation?

Documentation/block/io-controller.txt line 85

Quoted 7 lines
Documentation/block/io-controller.txt new file +7 -0
+- BFQ was designed to provide tighter bounds/delay w.r.t service provided
+  to a queue. Delay/Jitter with BFQ is supposed to be O(1).
+
+  Note: BFQ originally used amount of IO done (number of sectors) as notion
+        of service provided. IOW, it tried to provide fairness in terms of
+        actual IO done and not in terms of actual time disk access was
+	given to a queue.

I assume by sectors you mean the kernel sector size?

Quoted 14 lines
Documentation/block/io-controller.txt new file +14 -0
+
+	This patcheset modified BFQ to provide fairness in time domain because
+	that's what CFQ does. So idea was try not to deviate too much from
+	the CFQ behavior initially.
+
+	Providing fairness in time domain makes accounting trciky because
+	due to command queueing, at one time there might be multiple requests
+	from different queues and there is no easy way to find out how much
+	disk time actually was consumed by the requests of a particular
+	queue. More about this in comments in source code.
+
+So it is yet to be seen if changing to time domain still retains BFQ gurantees
+or not.
+

Documentation/block/io-controller.txt line 109

Quoted 10 lines
Documentation/block/io-controller.txt new file +10 -0
+From data structure point of view, one can think of a tree per device, where
+io groups and io queues are hanging and are being scheduled using B-WF2Q+
+algorithm. io_queue, is end queue where requests are actually stored and
+dispatched from (like cfqq).
+
+These io queues are primarily created by and managed by end io schedulers
+depending on its semantics. For example, noop, deadline and AS ioschedulers
+keep one io queues per cgroup and cfqq keeps one io queue per io_context in
+a cgroup (apart from async queues).
+

I assume there is one io_context per cgroup.

Documentation/block/io-controller.txt line 114

Quoted 5 lines
Documentation/block/io-controller.txt new file +5 -0
+A request is mapped to an io group by elevator layer and which io queue it
+is mapped to with in group depends on ioscheduler. Currently "current" task
+is used to determine the cgroup (hence io group) of the request. Down the
+line we need to make use of bio-cgroup patches to map delayed writes to
+right group.

That seem acceptable

Quoted 46 lines
Documentation/block/io-controller.txt new file +46 -0
+
+Going back to old behavior
+==========================
+In new scheme of things essentially we are creating hierarchical fair
+queuing logic in elevator layer and chaning IO schedulers to make use of
+that logic so that end IO schedulers start supporting hierarchical scheduling.
+
+Elevator layer continues to support the old interfaces. So even if fair queuing
+is enabled at elevator layer, one can have both new hierchical scheduler as
+well as old non-hierarchical scheduler operating.
+
+Also noop, deadline and AS have option of enabling hierarchical scheduling.
+If it is selected, fair queuing is done in hierarchical manner. If hierarchical
+scheduling is disabled, noop, deadline and AS should retain their existing
+behavior.
+
+CFQ is the only exception where one can not disable fair queuing as it is
+needed for provding fairness among various threads even in non-hierarchical
+mode.
+
+Various user visible config options
+===================================
+CONFIG_IOSCHED_NOOP_HIER
+	- Enables hierchical fair queuing in noop. Not selecting this option
+	  leads to old behavior of noop.
+
+CONFIG_IOSCHED_DEADLINE_HIER
+	- Enables hierchical fair queuing in deadline. Not selecting this
+	  option leads to old behavior of deadline.
+
+CONFIG_IOSCHED_AS_HIER
+	- Enables hierchical fair queuing in AS. Not selecting this option
+	  leads to old behavior of AS.
+
+CONFIG_IOSCHED_CFQ_HIER
+	- Enables hierarchical fair queuing in CFQ. Not selecting this option
+	  still does fair queuing among various queus but it is flat and not
+	  hierarchical.
+
+Config options selected automatically
+=====================================
+These config options are not user visible and are selected/deselected
+automatically based on IO scheduler configurations.
+
+CONFIG_ELV_FAIR_QUEUING
+	- Enables/Disables the fair queuing logic at elevator layer.

Documentation/block/io-controller.txt line 170

Quoted 10 lines
Documentation/block/io-controller.txt new file +10 -0
+
+CONFIG_GROUP_IOSCHED
+	- Enables/Disables hierarchical queuing and associated cgroup bits.
+
+TODO
+====
+- Lots of cleanups, testing, bug fixing, optimizations, benchmarking etc...
+- Convert cgroup ioprio to notion of weight.
+- Anticipatory code will need more work. It is not working properly currently
+  and needs more thought.

What are the problems with the code?

Documentation/block/io-controller.txt line 171

Quoted 1 line
Documentation/block/io-controller.txt new file +1 -0
+- Use of bio-cgroup patches.

I saw these posted as well

Documentation/block/io-controller.txt line 173

Quoted 2 lines
Documentation/block/io-controller.txt new file +2 -0
+- Use of Nauman's per cgroup request descriptor patches.
+

More details would be nice, I am not sure I understand

Quoted 9 lines
Documentation/block/io-controller.txt new file +9 -0
+HOWTO
+=====
+So far I have done very simple testing of running two dd threads in two
+different cgroups. Here is what you can do.
+
+- Enable hierarchical scheduling in io scheuduler of your choice (say cfq).
+	CONFIG_IOSCHED_CFQ_HIER=y
+
+- Compile and boot into kernel and mount IO controller.

Documentation/block/io-controller.txt line 192

Quoted 10 lines
Documentation/block/io-controller.txt new file +10 -0
+
+	mount -t cgroup -o io none /cgroup
+
+- Create two cgroups
+	mkdir -p /cgroup/test1/ /cgroup/test2
+
+- Set io priority of group test1 and test2
+	echo 0 > /cgroup/test1/io.ioprio
+	echo 4 > /cgroup/test2/io.ioprio
+

What is the meaning of priorities? Which is higher, which is lower? What is the maximum? How does it impact b/w?

Quoted 33 lines
Documentation/block/io-controller.txt new file +29 -0
+- Create two same size files (say 512MB each) on same disk (file1, file2) and
+  launch two dd threads in different cgroup to read those files. Make sure
+  right io scheduler is being used for the block device where files are
+  present (the one you compiled in hierarchical mode).
+
+	echo 1 > /proc/sys/vm/drop_caches
+
+	dd if=/mnt/lv0/zerofile1 of=/dev/null &
+	echo $! > /cgroup/test1/tasks
+	cat /cgroup/test1/tasks
+
+	dd if=/mnt/lv0/zerofile2 of=/dev/null &
+	echo $! > /cgroup/test2/tasks
+	cat /cgroup/test2/tasks
+
+- First dd should finish first.
+
+Some Test Results
+=================
+- Two dd in two cgroups with prio 0 and 4. Ran two "dd" in those cgroups.
+
+234179072 bytes (234 MB) copied, 10.1811 s, 23.0 MB/s
+234179072 bytes (234 MB) copied, 12.6187 s, 18.6 MB/s
+
+- Three dd in three cgroups with prio 0, 4, 4.
+
+234179072 bytes (234 MB) copied, 13.7654 s, 17.0 MB/s
+234179072 bytes (234 MB) copied, 19.476 s, 12.0 MB/s
+234179072 bytes (234 MB) copied, 20.1858 s, 11.6 MB/s
Signature · 1 line
1.6.0.1
Signature · 1 line
	Balbir
#97 lore raw

Re: [PATCH 01/10] Documentation

Quoted 212 lines · Balbir Singh
Quoted 49 lines · Vivek Goyal

o Documentation for io-controller.

---
 Documentation/block/io-controller.txt |  221 +++++++++++++++++++++++++++++++++

 1 files changed, 221 insertions(+), 0 deletions(-)

 create mode 100644 Documentation/block/io-controller.txt

Documentation/block/io-controller.txt new file +35 -0
new file mode 100644
index 0000000..8884c5a
@@ -0,0 +1,221 @@
+                             IO Controller
+                             =============
+
+Overview
+========
+
+This patchset implements a proportional weight IO controller. That is one
+can create cgroups and assign prio/weights to those cgroups and task group
+will get access to disk proportionate to the weight of the group.
+
+These patches modify elevator layer and individual IO schedulers to do
+IO control hence this io controller works only on block devices which use
+one of the standard io schedulers can not be used with any xyz logical block
+device.
+
+The assumption/thought behind modifying IO scheduler is that resource control
+is needed only on leaf nodes where the actual contention for resources is
+present and not on intertermediate logical block devices.
+
+Consider following hypothetical scenario. Lets say there are three physical
+disks, namely sda, sdb and sdc. Two logical volumes (lv0 and lv1) have been
+created on top of these. Some part of sdb is in lv0 and some part is in lv1.
+
+                         lv0      lv1
+                       /     \  /     \
+                     sda      sdb      sdc
+
+Also consider following cgroup hierarchy
+
+                             root
+                             /   \
+                            A     B
+                           / \    / \
+                          T1 T2  T3  T4
+

Documentation/block/io-controller.txt line 45

Quoted 10 lines
Documentation/block/io-controller.txt new file +10 -0
+A and B are two cgroups and T1, T2, T3 and T4 are tasks with-in those cgroups.
+Assuming T1, T2, T3 and T4 are doing IO on lv0 and lv1. These tasks should
+get their fair share of bandwidth on disks sda, sdb and sdc. There is no
+IO control on intermediate logical block nodes (lv0, lv1).
+
+So if tasks T1 and T2 are doing IO on lv0 and T3 and T4 are doing IO on lv1
+only, there will not be any contetion for resources between group A and B if
+IO is going to sda or sdc. But if actual IO gets translated to disk sdb, then
+IO scheduler associated with the sdb will distribute disk bandwidth to
+group A and B proportionate to their weight.

What if we have partitions sda1, sda2 and sda3 instead of sda, sdb and sdc?

Quoted 9 lines
Documentation/block/io-controller.txt new file +9 -0
+
+CFQ already has the notion of fairness and it provides differential disk
+access based on priority and class of the task. Just that it is flat and
+with cgroup stuff, it needs to be made hierarchical.
+
+Rest of the IO schedulers (noop, deadline and AS) don't have any notion
+of fairness among various threads.
+
+One of the concerns raised with modifying IO schedulers was that we don't

Documentation/block/io-controller.txt line 64

Quoted 10 lines
Documentation/block/io-controller.txt new file +10 -0
+want to replicate the code in all the IO schedulers. These patches share
+the fair queuing code which has been moved to a common layer (elevator
+layer). Hence we don't end up replicating code across IO schedulers.
+
+Design
+======
+This patchset primarily uses BFQ (Budget Fair Queuing) code to provide
+fairness among different IO queues. Fabio and Paolo implemented BFQ which uses
+B-WF2Q+ algorithm for fair queuing.
+

References to BFQ, please. I can search them, but having them in the doc would be nice.

Quoted 4 lines
Documentation/block/io-controller.txt new file +4 -0
+Why BFQ?
+
+- Not sure if weighted round robin logic of CFQ can be easily extended for
+  hierarchical mode. One of the things is that we can not keep dividing

Documentation/block/io-controller.txt line 78

Quoted 10 lines
Documentation/block/io-controller.txt new file +10 -0
+  the time slice of parent group among childrens. Deeper we go in hierarchy
+  time slice will get smaller.
+
+  One of the ways to implement hierarchical support could be to keep track
+  of virtual time and service provided to queue/group and select a queue/group
+  for service based on any of the various available algoriths.
+
+  BFQ already had support for hierarchical scheduling, taking those patches
+  was easier.
+

Could you elaborate, when you say timeslices get smaller -

1. Are you referring to inability to use higher resolution time?
2. Loss of throughput due to timeslice degradation?

Documentation/block/io-controller.txt line 85

Quoted 7 lines
Documentation/block/io-controller.txt new file +7 -0
+- BFQ was designed to provide tighter bounds/delay w.r.t service provided
+  to a queue. Delay/Jitter with BFQ is supposed to be O(1).
+
+  Note: BFQ originally used amount of IO done (number of sectors) as notion
+        of service provided. IOW, it tried to provide fairness in terms of
+        actual IO done and not in terms of actual time disk access was
+     given to a queue.

I assume by sectors you mean the kernel sector size?

Quoted 14 lines
Documentation/block/io-controller.txt new file +14 -0
+
+     This patcheset modified BFQ to provide fairness in time domain because
+     that's what CFQ does. So idea was try not to deviate too much from
+     the CFQ behavior initially.
+
+     Providing fairness in time domain makes accounting trciky because
+     due to command queueing, at one time there might be multiple requests
+     from different queues and there is no easy way to find out how much
+     disk time actually was consumed by the requests of a particular
+     queue. More about this in comments in source code.
+
+So it is yet to be seen if changing to time domain still retains BFQ gurantees
+or not.
+

Documentation/block/io-controller.txt line 109

Quoted 10 lines
Documentation/block/io-controller.txt new file +10 -0
+From data structure point of view, one can think of a tree per device, where
+io groups and io queues are hanging and are being scheduled using B-WF2Q+
+algorithm. io_queue, is end queue where requests are actually stored and
+dispatched from (like cfqq).
+
+These io queues are primarily created by and managed by end io schedulers
+depending on its semantics. For example, noop, deadline and AS ioschedulers
+keep one io queues per cgroup and cfqq keeps one io queue per io_context in
+a cgroup (apart from async queues).
+

I assume there is one io_context per cgroup.

Documentation/block/io-controller.txt line 114

Quoted 5 lines
Documentation/block/io-controller.txt new file +5 -0
+A request is mapped to an io group by elevator layer and which io queue it
+is mapped to with in group depends on ioscheduler. Currently "current" task
+is used to determine the cgroup (hence io group) of the request. Down the
+line we need to make use of bio-cgroup patches to map delayed writes to
+right group.

That seem acceptable

Quoted 46 lines
Documentation/block/io-controller.txt new file +46 -0
+
+Going back to old behavior
+==========================
+In new scheme of things essentially we are creating hierarchical fair
+queuing logic in elevator layer and chaning IO schedulers to make use of
+that logic so that end IO schedulers start supporting hierarchical scheduling.
+
+Elevator layer continues to support the old interfaces. So even if fair queuing
+is enabled at elevator layer, one can have both new hierchical scheduler as
+well as old non-hierarchical scheduler operating.
+
+Also noop, deadline and AS have option of enabling hierarchical scheduling.
+If it is selected, fair queuing is done in hierarchical manner. If hierarchical
+scheduling is disabled, noop, deadline and AS should retain their existing
+behavior.
+
+CFQ is the only exception where one can not disable fair queuing as it is
+needed for provding fairness among various threads even in non-hierarchical
+mode.
+
+Various user visible config options
+===================================
+CONFIG_IOSCHED_NOOP_HIER
+     - Enables hierchical fair queuing in noop. Not selecting this option
+       leads to old behavior of noop.
+
+CONFIG_IOSCHED_DEADLINE_HIER
+     - Enables hierchical fair queuing in deadline. Not selecting this
+       option leads to old behavior of deadline.
+
+CONFIG_IOSCHED_AS_HIER
+     - Enables hierchical fair queuing in AS. Not selecting this option
+       leads to old behavior of AS.
+
+CONFIG_IOSCHED_CFQ_HIER
+     - Enables hierarchical fair queuing in CFQ. Not selecting this option
+       still does fair queuing among various queus but it is flat and not
+       hierarchical.
+
+Config options selected automatically
+=====================================
+These config options are not user visible and are selected/deselected
+automatically based on IO scheduler configurations.
+
+CONFIG_ELV_FAIR_QUEUING
+     - Enables/Disables the fair queuing logic at elevator layer.

Documentation/block/io-controller.txt line 170

Quoted 10 lines
Documentation/block/io-controller.txt new file +10 -0
+
+CONFIG_GROUP_IOSCHED
+     - Enables/Disables hierarchical queuing and associated cgroup bits.
+
+TODO
+====
+- Lots of cleanups, testing, bug fixing, optimizations, benchmarking etc...
+- Convert cgroup ioprio to notion of weight.
+- Anticipatory code will need more work. It is not working properly currently
+  and needs more thought.

What are the problems with the code?

Quoted 3 lines

Documentation/block/io-controller.txt line 171

Quoted 1 line
Documentation/block/io-controller.txt new file +1 -0
+- Use of bio-cgroup patches.

I saw these posted as well

I have refactored the bio-cgroup patches to work on top of this patch set, and keep track of async writes. But we have not been able to get proportional division for async writes. The problem seems to stem from the fact that pdflush is cgroup agnostic. Getting proportional IO scheduling to work might need work beyond block layer. Vivek has been able to do more testing with those patches, and can explain more.

Quoted 5 lines

Documentation/block/io-controller.txt line 173

Quoted 3 lines
Documentation/block/io-controller.txt new file +2 -0
+- Use of Nauman's per cgroup request descriptor patches.
+

More details would be nice, I am not sure I understand

Right now, the block layer has a limit on request descriptors that can be allocated. Once that limit is reached, a process trying to get a request descriptor would be blocked. I wrote a patch in which I made the request descriptor limit per cgroup, i.e a process will only be blocked if request descriptors allocated to a give cgroup exceed a certain limit.

This patch set is already big and we are trying to be careful to include all the work we have done for solving the problem. So I was planning to hold onto that patch, and send it out for comments once the basic infrastructure gets some traction.

Quoted 61 lines
Quoted 10 lines
Documentation/block/io-controller.txt new file +9 -0
+HOWTO
+=====
+So far I have done very simple testing of running two dd threads in two
+different cgroups. Here is what you can do.
+
+- Enable hierarchical scheduling in io scheuduler of your choice (say cfq).
+     CONFIG_IOSCHED_CFQ_HIER=y
+
+- Compile and boot into kernel and mount IO controller.

Documentation/block/io-controller.txt line 192

Quoted 10 lines
Documentation/block/io-controller.txt new file +10 -0
+
+     mount -t cgroup -o io none /cgroup
+
+- Create two cgroups
+     mkdir -p /cgroup/test1/ /cgroup/test2
+
+- Set io priority of group test1 and test2
+     echo 0 > /cgroup/test1/io.ioprio
+     echo 4 > /cgroup/test2/io.ioprio
+

What is the meaning of priorities? Which is higher, which is lower? What is the maximum? How does it impact b/w?

Quoted 30 lines
Documentation/block/io-controller.txt new file +29 -1
+- Create two same size files (say 512MB each) on same disk (file1, file2) and
+  launch two dd threads in different cgroup to read those files. Make sure
+  right io scheduler is being used for the block device where files are
+  present (the one you compiled in hierarchical mode).
+
+     echo 1 > /proc/sys/vm/drop_caches
+
+     dd if=/mnt/lv0/zerofile1 of=/dev/null &
+     echo $! > /cgroup/test1/tasks
+     cat /cgroup/test1/tasks
+
+     dd if=/mnt/lv0/zerofile2 of=/dev/null &
+     echo $! > /cgroup/test2/tasks
+     cat /cgroup/test2/tasks
+
+- First dd should finish first.
+
+Some Test Results
+=================
+- Two dd in two cgroups with prio 0 and 4. Ran two "dd" in those cgroups.
+
+234179072 bytes (234 MB) copied, 10.1811 s, 23.0 MB/s
+234179072 bytes (234 MB) copied, 12.6187 s, 18.6 MB/s
+
+- Three dd in three cgroups with prio 0, 4, 4.
+
+234179072 bytes (234 MB) copied, 13.7654 s, 17.0 MB/s
+234179072 bytes (234 MB) copied, 19.476 s, 12.0 MB/s
+234179072 bytes (234 MB) copied, 20.1858 s, 11.6 MB/s
--
Quoted 3 lines

1.6.0.1

--
       Balbir

A message that is not in the archive (sent off-list or not yet received): 20090406143556.GK7082-SINUvgVNF2CyUtPGxGje5AC/[email protected]

#99 lore raw

Re: [PATCH 01/10] Documentation

Quoted 20 lines · Balbir Singh
Quoted 18 lines · Vivek Goyal
Documentation/block/io-controller.txt new file +18 -0
+
+			    lv0      lv1
+			  /	\  /     \
+			sda      sdb      sdc
+
+Also consider following cgroup hierarchy
+
+				root
+				/   \
+			       A     B
+			      / \    / \
+			     T1 T2  T3  T4
+
+A and B are two cgroups and T1, T2, T3 and T4 are tasks with-in those cgroups.
+Assuming T1, T2, T3 and T4 are doing IO on lv0 and lv1. These tasks should
+get their fair share of bandwidth on disks sda, sdb and sdc. There is no
+IO control on intermediate logical block nodes (lv0, lv1).
+
Quoted 8 lines

Documentation/block/io-controller.txt line 45

Quoted 5 lines
Documentation/block/io-controller.txt new file +5 -0
+So if tasks T1 and T2 are doing IO on lv0 and T3 and T4 are doing IO on lv1
+only, there will not be any contetion for resources between group A and B if
+IO is going to sda or sdc. But if actual IO gets translated to disk sdb, then
+IO scheduler associated with the sdb will distribute disk bandwidth to
+group A and B proportionate to their weight.

What if we have partitions sda1, sda2 and sda3 instead of sda, sdb and sdc?

The bandwidth controlling is device basis, so with sda1, sda2 and sda3 instead, they will contending on sda.

Quoted 139 lines
Quoted 10 lines
Documentation/block/io-controller.txt new file +9 -0
+
+CFQ already has the notion of fairness and it provides differential disk
+access based on priority and class of the task. Just that it is flat and
+with cgroup stuff, it needs to be made hierarchical.
+
+Rest of the IO schedulers (noop, deadline and AS) don't have any notion
+of fairness among various threads.
+
+One of the concerns raised with modifying IO schedulers was that we don't

Documentation/block/io-controller.txt line 64

Quoted 10 lines
Documentation/block/io-controller.txt new file +10 -0
+want to replicate the code in all the IO schedulers. These patches share
+the fair queuing code which has been moved to a common layer (elevator
+layer). Hence we don't end up replicating code across IO schedulers.
+
+Design
+======
+This patchset primarily uses BFQ (Budget Fair Queuing) code to provide
+fairness among different IO queues. Fabio and Paolo implemented BFQ which uses
+B-WF2Q+ algorithm for fair queuing.
+

References to BFQ, please. I can search them, but having them in the doc would be nice.

Quoted 4 lines
Documentation/block/io-controller.txt new file +4 -0
+Why BFQ?
+
+- Not sure if weighted round robin logic of CFQ can be easily extended for
+  hierarchical mode. One of the things is that we can not keep dividing

Documentation/block/io-controller.txt line 78

Quoted 10 lines
Documentation/block/io-controller.txt new file +10 -0
+  the time slice of parent group among childrens. Deeper we go in hierarchy
+  time slice will get smaller.
+
+  One of the ways to implement hierarchical support could be to keep track
+  of virtual time and service provided to queue/group and select a queue/group
+  for service based on any of the various available algoriths.
+
+  BFQ already had support for hierarchical scheduling, taking those patches
+  was easier.
+

Could you elaborate, when you say timeslices get smaller -

1. Are you referring to inability to use higher resolution time?
2. Loss of throughput due to timeslice degradation?

Documentation/block/io-controller.txt line 85

Quoted 7 lines

+- BFQ was designed to provide tighter bounds/delay w.r.t service provided

Documentation/block/io-controller.txt new file +6 -0
+  to a queue. Delay/Jitter with BFQ is supposed to be O(1).
+
+  Note: BFQ originally used amount of IO done (number of sectors) as notion
+        of service provided. IOW, it tried to provide fairness in terms of
+        actual IO done and not in terms of actual time disk access was
+	given to a queue.

I assume by sectors you mean the kernel sector size?

Quoted 14 lines
Documentation/block/io-controller.txt new file +14 -0
+
+	This patcheset modified BFQ to provide fairness in time domain because
+	that's what CFQ does. So idea was try not to deviate too much from
+	the CFQ behavior initially.
+
+	Providing fairness in time domain makes accounting trciky because
+	due to command queueing, at one time there might be multiple requests
+	from different queues and there is no easy way to find out how much
+	disk time actually was consumed by the requests of a particular
+	queue. More about this in comments in source code.
+
+So it is yet to be seen if changing to time domain still retains BFQ gurantees
+or not.
+

Documentation/block/io-controller.txt line 109

Quoted 10 lines
Documentation/block/io-controller.txt new file +10 -0
+From data structure point of view, one can think of a tree per device, where
+io groups and io queues are hanging and are being scheduled using B-WF2Q+
+algorithm. io_queue, is end queue where requests are actually stored and
+dispatched from (like cfqq).
+
+These io queues are primarily created by and managed by end io schedulers
+depending on its semantics. For example, noop, deadline and AS ioschedulers
+keep one io queues per cgroup and cfqq keeps one io queue per io_context in
+a cgroup (apart from async queues).
+

I assume there is one io_context per cgroup.

Documentation/block/io-controller.txt line 114

Quoted 5 lines
Documentation/block/io-controller.txt new file +5 -0
+A request is mapped to an io group by elevator layer and which io queue it
+is mapped to with in group depends on ioscheduler. Currently "current" task
+is used to determine the cgroup (hence io group) of the request. Down the
+line we need to make use of bio-cgroup patches to map delayed writes to
+right group.

That seem acceptable

Quoted 50 lines
Documentation/block/io-controller.txt new file +50 -0
+
+Going back to old behavior
+==========================
+In new scheme of things essentially we are creating hierarchical fair
+queuing logic in elevator layer and chaning IO schedulers to make use of
+that logic so that end IO schedulers start supporting hierarchical scheduling.
+
+Elevator layer continues to support the old interfaces. So even if fair queuing
+is enabled at elevator layer, one can have both new hierchical scheduler as
+well as old non-hierarchical scheduler operating.
+
+Also noop, deadline and AS have option of enabling hierarchical scheduling.
+If it is selected, fair queuing is done in hierarchical manner. If hierarchical
+scheduling is disabled, noop, deadline and AS should retain their existing
+behavior.
+
+CFQ is the only exception where one can not disable fair queuing as it is
+needed for provding fairness among various threads even in non-hierarchical
+mode.
+
+Various user visible config options
+===================================
+CONFIG_IOSCHED_NOOP_HIER
+	- Enables hierchical fair queuing in noop. Not selecting this option
+	  leads to old behavior of noop.
+
+CONFIG_IOSCHED_DEADLINE_HIER
+	- Enables hierchical fair queuing in deadline. Not selecting this
+	  option leads to old behavior of deadline.
+
+CONFIG_IOSCHED_AS_HIER
+	- Enables hierchical fair queuing in AS. Not selecting this option
+	  leads to old behavior of AS.
+
+CONFIG_IOSCHED_CFQ_HIER
+	- Enables hierarchical fair queuing in CFQ. Not selecting this option
+	  still does fair queuing among various queus but it is flat and not
+	  hierarchical.
+
+Config options selected automatically
+=====================================
+These config options are not user visible and are selected/deselected
+automatically based on IO scheduler configurations.
+
+CONFIG_ELV_FAIR_QUEUING
+	- Enables/Disables the fair queuing logic at elevator layer.
+
+CONFIG_GROUP_IOSCHED
+	- Enables/Disables hierarchical queuing and associated cgroup bits.
+
Quoted 8 lines

Documentation/block/io-controller.txt line 170

Quoted 6 lines
Documentation/block/io-controller.txt new file +6 -0
+TODO
+====
+- Lots of cleanups, testing, bug fixing, optimizations, benchmarking etc...
+- Convert cgroup ioprio to notion of weight.
+- Anticipatory code will need more work. It is not working properly currently
+  and needs more thought.

What are the problems with the code?

Anticipatory has its own idling logic, so what is the concerning here is how to make as work together with commom layer.

Signature · 2 lines
Regards
Gui Jianfeng
#100 lore raw

Re: [PATCH 01/10] Documentation

In reply to a message not in the archive
Quoted 2 lines · Balbir Singh

* Vivek Goyal <[email protected]> [2009-03-11 21:56:46]:

Thanks for having a look balbir. Sorry for the late reply..

[..]

Quoted 21 lines
Quoted 21 lines
Documentation/block/io-controller.txt new file +21 -0
+Consider following hypothetical scenario. Lets say there are three physical
+disks, namely sda, sdb and sdc. Two logical volumes (lv0 and lv1) have been
+created on top of these. Some part of sdb is in lv0 and some part is in lv1.
+
+			    lv0      lv1
+			  /	\  /     \
+			sda      sdb      sdc
+
+Also consider following cgroup hierarchy
+
+				root
+				/   \
+			       A     B
+			      / \    / \
+			     T1 T2  T3  T4
+
+A and B are two cgroups and T1, T2, T3 and T4 are tasks with-in those cgroups.
+Assuming T1, T2, T3 and T4 are doing IO on lv0 and lv1. These tasks should
+get their fair share of bandwidth on disks sda, sdb and sdc. There is no
+IO control on intermediate logical block nodes (lv0, lv1).
+
Quoted 8 lines

Documentation/block/io-controller.txt line 45

Quoted 5 lines
Documentation/block/io-controller.txt new file +5 -0
+So if tasks T1 and T2 are doing IO on lv0 and T3 and T4 are doing IO on lv1
+only, there will not be any contetion for resources between group A and B if
+IO is going to sda or sdc. But if actual IO gets translated to disk sdb, then
+IO scheduler associated with the sdb will distribute disk bandwidth to
+group A and B proportionate to their weight.

What if we have partitions sda1, sda2 and sda3 instead of sda, sdb and sdc?

As Gui already mentioned, IO control is on per device basis (like IO scheduler) and we don't try to control it per partition basis.

Quoted 15 lines
Quoted 15 lines
Documentation/block/io-controller.txt new file +14 -0
+
+CFQ already has the notion of fairness and it provides differential disk
+access based on priority and class of the task. Just that it is flat and
+with cgroup stuff, it needs to be made hierarchical.
+
+Rest of the IO schedulers (noop, deadline and AS) don't have any notion
+of fairness among various threads.
+
+One of the concerns raised with modifying IO schedulers was that we don't
+want to replicate the code in all the IO schedulers. These patches share
+the fair queuing code which has been moved to a common layer (elevator
+layer). Hence we don't end up replicating code across IO schedulers.
+
+Design
Quoted 8 lines

Documentation/block/io-controller.txt line 64

Quoted 5 lines

+======

Documentation/block/io-controller.txt new file +4 -0
+This patchset primarily uses BFQ (Budget Fair Queuing) code to provide
+fairness among different IO queues. Fabio and Paolo implemented BFQ which uses
+B-WF2Q+ algorithm for fair queuing.
+

References to BFQ, please. I can search them, but having them in the doc would be nice.

That's a good point. In next posting I will put references also.

Quoted 18 lines
Quoted 5 lines
Documentation/block/io-controller.txt new file +4 -0
+Why BFQ?
+
+- Not sure if weighted round robin logic of CFQ can be easily extended for
+  hierarchical mode. One of the things is that we can not keep dividing

Documentation/block/io-controller.txt line 78

Quoted 10 lines
Documentation/block/io-controller.txt new file +10 -0
+  the time slice of parent group among childrens. Deeper we go in hierarchy
+  time slice will get smaller.
+
+  One of the ways to implement hierarchical support could be to keep track
+  of virtual time and service provided to queue/group and select a queue/group
+  for service based on any of the various available algoriths.
+
+  BFQ already had support for hierarchical scheduling, taking those patches
+  was easier.
+

Could you elaborate, when you say timeslices get smaller -

Quoted 2 lines

1. Are you referring to inability to use higher resolution time?
2. Loss of throughput due to timeslice degradation?

I think keeping a track of time using higher resolution time should not be a problem but it would be rather more of loss of throughput due to smaller timeslices and frequent queue switching.

Quoted 10 lines

Documentation/block/io-controller.txt line 85

Quoted 8 lines

+- BFQ was designed to provide tighter bounds/delay w.r.t service provided

Documentation/block/io-controller.txt new file +6 -0
+  to a queue. Delay/Jitter with BFQ is supposed to be O(1).
+
+  Note: BFQ originally used amount of IO done (number of sectors) as notion
+        of service provided. IOW, it tried to provide fairness in terms of
+        actual IO done and not in terms of actual time disk access was
+	given to a queue.

I assume by sectors you mean the kernel sector size?

Yes.

Quoted 19 lines
Quoted 19 lines
Documentation/block/io-controller.txt new file +18 -0
+
+	This patcheset modified BFQ to provide fairness in time domain because
+	that's what CFQ does. So idea was try not to deviate too much from
+	the CFQ behavior initially.
+
+	Providing fairness in time domain makes accounting trciky because
+	due to command queueing, at one time there might be multiple requests
+	from different queues and there is no easy way to find out how much
+	disk time actually was consumed by the requests of a particular
+	queue. More about this in comments in source code.
+
+So it is yet to be seen if changing to time domain still retains BFQ gurantees
+or not.
+
+From data structure point of view, one can think of a tree per device, where
+io groups and io queues are hanging and are being scheduled using B-WF2Q+
+algorithm. io_queue, is end queue where requests are actually stored and
+dispatched from (like cfqq).
Quoted 8 lines

Documentation/block/io-controller.txt line 109

Quoted 6 lines
Documentation/block/io-controller.txt new file +6 -0
+
+These io queues are primarily created by and managed by end io schedulers
+depending on its semantics. For example, noop, deadline and AS ioschedulers
+keep one io queues per cgroup and cfqq keeps one io queue per io_context in
+a cgroup (apart from async queues).
+

I assume there is one io_context per cgroup.

No. There can be multiple io_context per cgroup because currently io_context is defined as threads which are doing IO sharing and are kept in one queue from IO point of view by cfq and multiple queues are not created. So there might be many processes/threads in a cgroup and not necessarily they are sharing the io_context.

Quoted 8 lines

Documentation/block/io-controller.txt line 114

Quoted 6 lines
Documentation/block/io-controller.txt new file +5 -0
+A request is mapped to an io group by elevator layer and which io queue it
+is mapped to with in group depends on ioscheduler. Currently "current" task
+is used to determine the cgroup (hence io group) of the request. Down the
+line we need to make use of bio-cgroup patches to map delayed writes to
+right group.

That seem acceptable

Andrew first wants to see a solid plan for handling async writes :-) So currently I am playing with patches to map writes to correct cgroup. Mapping the IO to right cgroup is only one part of the problem. Other part is that I am not seeing a continuious stream of writes at IO scheduler level. So if two dd processes are running in user space, ideally one can expect two continuous stream of write requests at IO scheduler but instead I see bursty serialized traffic. So a bunch of write request from first dd then another bunch of write requests from second dd and it goes on.. and this leads to no service differentiation between two writes because when higher priority task is not dispatching any IO (for .2 seconds), lower priority task/group gets to use full disk and soon catches up with higher priority one..

Part of this serialization was taking place at request descriptor allocation infrastructure where number of request descriptors are limited and if one writer first consumes most of the descriptors it will block/serialize other writer.

Now I have got a crude working patch where I can limit per group request descriptors so that one group can not block other group. But still don't see continuously backlogged write queues at IO scheduler...

Time to do more debugging and move up the layer and see where this serialization is taking place (i guess page cache...).

Quoted 51 lines
Quoted 51 lines
Documentation/block/io-controller.txt new file +50 -0
+
+Going back to old behavior
+==========================
+In new scheme of things essentially we are creating hierarchical fair
+queuing logic in elevator layer and chaning IO schedulers to make use of
+that logic so that end IO schedulers start supporting hierarchical scheduling.
+
+Elevator layer continues to support the old interfaces. So even if fair queuing
+is enabled at elevator layer, one can have both new hierchical scheduler as
+well as old non-hierarchical scheduler operating.
+
+Also noop, deadline and AS have option of enabling hierarchical scheduling.
+If it is selected, fair queuing is done in hierarchical manner. If hierarchical
+scheduling is disabled, noop, deadline and AS should retain their existing
+behavior.
+
+CFQ is the only exception where one can not disable fair queuing as it is
+needed for provding fairness among various threads even in non-hierarchical
+mode.
+
+Various user visible config options
+===================================
+CONFIG_IOSCHED_NOOP_HIER
+	- Enables hierchical fair queuing in noop. Not selecting this option
+	  leads to old behavior of noop.
+
+CONFIG_IOSCHED_DEADLINE_HIER
+	- Enables hierchical fair queuing in deadline. Not selecting this
+	  option leads to old behavior of deadline.
+
+CONFIG_IOSCHED_AS_HIER
+	- Enables hierchical fair queuing in AS. Not selecting this option
+	  leads to old behavior of AS.
+
+CONFIG_IOSCHED_CFQ_HIER
+	- Enables hierarchical fair queuing in CFQ. Not selecting this option
+	  still does fair queuing among various queus but it is flat and not
+	  hierarchical.
+
+Config options selected automatically
+=====================================
+These config options are not user visible and are selected/deselected
+automatically based on IO scheduler configurations.
+
+CONFIG_ELV_FAIR_QUEUING
+	- Enables/Disables the fair queuing logic at elevator layer.
+
+CONFIG_GROUP_IOSCHED
+	- Enables/Disables hierarchical queuing and associated cgroup bits.
+
Quoted 8 lines

Documentation/block/io-controller.txt line 170

Quoted 6 lines
Documentation/block/io-controller.txt new file +6 -0
+TODO
+====
+- Lots of cleanups, testing, bug fixing, optimizations, benchmarking etc...
+- Convert cgroup ioprio to notion of weight.
+- Anticipatory code will need more work. It is not working properly currently
+  and needs more thought.

What are the problems with the code?

Have not got a chance to look into the issues in detail yet. Just a crude run saw drop in performance. Will debug it later the moment I have got async writes handled...

Quoted 8 lines
Quoted 1 line

+- Use of bio-cgroup patches.

I saw these posted as well

Quoted 2 lines

+- Use of Nauman's per cgroup request descriptor patches. +

More details would be nice, I am not sure I understand

Currently the number of request descriptors which can be allocated per device/request queue are fixed by a sysfs tunable (q->nr_requests). So if there is lots of IO going on from one cgroup then it will consume all the available request descriptors and other cgroup might starve and not get its fair share.

Hence we also need to introduce the notion of request descriptor limit per cgroup so that if request descriptors from one group are exhausted, then it does not impact the IO of other cgroup.

Quoted 15 lines
Quoted 15 lines
Documentation/block/io-controller.txt new file +14 -0
+HOWTO
+=====
+So far I have done very simple testing of running two dd threads in two
+different cgroups. Here is what you can do.
+
+- Enable hierarchical scheduling in io scheuduler of your choice (say cfq).
+	CONFIG_IOSCHED_CFQ_HIER=y
+
+- Compile and boot into kernel and mount IO controller.
+
+	mount -t cgroup -o io none /cgroup
+
+- Create two cgroups
+	mkdir -p /cgroup/test1/ /cgroup/test2
Quoted 8 lines

Documentation/block/io-controller.txt line 192

Quoted 5 lines
Documentation/block/io-controller.txt new file +5 -0
+
+- Set io priority of group test1 and test2
+	echo 0 > /cgroup/test1/io.ioprio
+	echo 4 > /cgroup/test2/io.ioprio
+

What is the meaning of priorities? Which is higher, which is lower? What is the maximum? How does it impact b/w?

Currently cfq has notion of priority range 0-7 (0 being highest). To being with we simply adopted that notion though we are converting it to weights now for group.

Mapping from group priority to group weight is linear. So prio 0 group should get double the BW of prio 4 group.

Thanks
Vivek

#101 lore raw

Re: [PATCH 01/10] Documentation

In reply to Balbir Singh
Quoted 20 lines · Balbir Singh
Quoted 18 lines · Vivek Goyal
Documentation/block/io-controller.txt new file +18 -0
+
+			    lv0      lv1
+			  /	\  /     \
+			sda      sdb      sdc
+
+Also consider following cgroup hierarchy
+
+				root
+				/   \
+			       A     B
+			      / \    / \
+			     T1 T2  T3  T4
+
+A and B are two cgroups and T1, T2, T3 and T4 are tasks with-in those cgroups.
+Assuming T1, T2, T3 and T4 are doing IO on lv0 and lv1. These tasks should
+get their fair share of bandwidth on disks sda, sdb and sdc. There is no
+IO control on intermediate logical block nodes (lv0, lv1).
+
Quoted 8 lines

Documentation/block/io-controller.txt line 45

Quoted 5 lines
Documentation/block/io-controller.txt new file +5 -0
+So if tasks T1 and T2 are doing IO on lv0 and T3 and T4 are doing IO on lv1
+only, there will not be any contetion for resources between group A and B if
+IO is going to sda or sdc. But if actual IO gets translated to disk sdb, then
+IO scheduler associated with the sdb will distribute disk bandwidth to
+group A and B proportionate to their weight.

What if we have partitions sda1, sda2 and sda3 instead of sda, sdb and sdc?

The bandwidth controlling is device basis, so with sda1, sda2 and sda3 instead, they will contending on sda.

Quoted 139 lines
Quoted 10 lines
Documentation/block/io-controller.txt new file +9 -0
+
+CFQ already has the notion of fairness and it provides differential disk
+access based on priority and class of the task. Just that it is flat and
+with cgroup stuff, it needs to be made hierarchical.
+
+Rest of the IO schedulers (noop, deadline and AS) don't have any notion
+of fairness among various threads.
+
+One of the concerns raised with modifying IO schedulers was that we don't

Documentation/block/io-controller.txt line 64

Quoted 10 lines
Documentation/block/io-controller.txt new file +10 -0
+want to replicate the code in all the IO schedulers. These patches share
+the fair queuing code which has been moved to a common layer (elevator
+layer). Hence we don't end up replicating code across IO schedulers.
+
+Design
+======
+This patchset primarily uses BFQ (Budget Fair Queuing) code to provide
+fairness among different IO queues. Fabio and Paolo implemented BFQ which uses
+B-WF2Q+ algorithm for fair queuing.
+

References to BFQ, please. I can search them, but having them in the doc would be nice.

Quoted 4 lines
Documentation/block/io-controller.txt new file +4 -0
+Why BFQ?
+
+- Not sure if weighted round robin logic of CFQ can be easily extended for
+  hierarchical mode. One of the things is that we can not keep dividing

Documentation/block/io-controller.txt line 78

Quoted 10 lines
Documentation/block/io-controller.txt new file +10 -0
+  the time slice of parent group among childrens. Deeper we go in hierarchy
+  time slice will get smaller.
+
+  One of the ways to implement hierarchical support could be to keep track
+  of virtual time and service provided to queue/group and select a queue/group
+  for service based on any of the various available algoriths.
+
+  BFQ already had support for hierarchical scheduling, taking those patches
+  was easier.
+

Could you elaborate, when you say timeslices get smaller -

1. Are you referring to inability to use higher resolution time?
2. Loss of throughput due to timeslice degradation?

Documentation/block/io-controller.txt line 85

Quoted 7 lines

+- BFQ was designed to provide tighter bounds/delay w.r.t service provided

Documentation/block/io-controller.txt new file +6 -0
+  to a queue. Delay/Jitter with BFQ is supposed to be O(1).
+
+  Note: BFQ originally used amount of IO done (number of sectors) as notion
+        of service provided. IOW, it tried to provide fairness in terms of
+        actual IO done and not in terms of actual time disk access was
+	given to a queue.

I assume by sectors you mean the kernel sector size?

Quoted 14 lines
Documentation/block/io-controller.txt new file +14 -0
+
+	This patcheset modified BFQ to provide fairness in time domain because
+	that's what CFQ does. So idea was try not to deviate too much from
+	the CFQ behavior initially.
+
+	Providing fairness in time domain makes accounting trciky because
+	due to command queueing, at one time there might be multiple requests
+	from different queues and there is no easy way to find out how much
+	disk time actually was consumed by the requests of a particular
+	queue. More about this in comments in source code.
+
+So it is yet to be seen if changing to time domain still retains BFQ gurantees
+or not.
+

Documentation/block/io-controller.txt line 109

Quoted 10 lines
Documentation/block/io-controller.txt new file +10 -0
+From data structure point of view, one can think of a tree per device, where
+io groups and io queues are hanging and are being scheduled using B-WF2Q+
+algorithm. io_queue, is end queue where requests are actually stored and
+dispatched from (like cfqq).
+
+These io queues are primarily created by and managed by end io schedulers
+depending on its semantics. For example, noop, deadline and AS ioschedulers
+keep one io queues per cgroup and cfqq keeps one io queue per io_context in
+a cgroup (apart from async queues).
+

I assume there is one io_context per cgroup.

Documentation/block/io-controller.txt line 114

Quoted 5 lines
Documentation/block/io-controller.txt new file +5 -0
+A request is mapped to an io group by elevator layer and which io queue it
+is mapped to with in group depends on ioscheduler. Currently "current" task
+is used to determine the cgroup (hence io group) of the request. Down the
+line we need to make use of bio-cgroup patches to map delayed writes to
+right group.

That seem acceptable

Quoted 50 lines
Documentation/block/io-controller.txt new file +50 -0
+
+Going back to old behavior
+==========================
+In new scheme of things essentially we are creating hierarchical fair
+queuing logic in elevator layer and chaning IO schedulers to make use of
+that logic so that end IO schedulers start supporting hierarchical scheduling.
+
+Elevator layer continues to support the old interfaces. So even if fair queuing
+is enabled at elevator layer, one can have both new hierchical scheduler as
+well as old non-hierarchical scheduler operating.
+
+Also noop, deadline and AS have option of enabling hierarchical scheduling.
+If it is selected, fair queuing is done in hierarchical manner. If hierarchical
+scheduling is disabled, noop, deadline and AS should retain their existing
+behavior.
+
+CFQ is the only exception where one can not disable fair queuing as it is
+needed for provding fairness among various threads even in non-hierarchical
+mode.
+
+Various user visible config options
+===================================
+CONFIG_IOSCHED_NOOP_HIER
+	- Enables hierchical fair queuing in noop. Not selecting this option
+	  leads to old behavior of noop.
+
+CONFIG_IOSCHED_DEADLINE_HIER
+	- Enables hierchical fair queuing in deadline. Not selecting this
+	  option leads to old behavior of deadline.
+
+CONFIG_IOSCHED_AS_HIER
+	- Enables hierchical fair queuing in AS. Not selecting this option
+	  leads to old behavior of AS.
+
+CONFIG_IOSCHED_CFQ_HIER
+	- Enables hierarchical fair queuing in CFQ. Not selecting this option
+	  still does fair queuing among various queus but it is flat and not
+	  hierarchical.
+
+Config options selected automatically
+=====================================
+These config options are not user visible and are selected/deselected
+automatically based on IO scheduler configurations.
+
+CONFIG_ELV_FAIR_QUEUING
+	- Enables/Disables the fair queuing logic at elevator layer.
+
+CONFIG_GROUP_IOSCHED
+	- Enables/Disables hierarchical queuing and associated cgroup bits.
+
Quoted 8 lines

Documentation/block/io-controller.txt line 170

Quoted 6 lines
Documentation/block/io-controller.txt new file +6 -0
+TODO
+====
+- Lots of cleanups, testing, bug fixing, optimizations, benchmarking etc...
+- Convert cgroup ioprio to notion of weight.
+- Anticipatory code will need more work. It is not working properly currently
+  and needs more thought.

What are the problems with the code?

Anticipatory has its own idling logic, so what is the concerning here is how to make as work together with commom layer.

Signature · 2 lines
Regards
Gui Jianfeng
#102 lore raw

Re: [PATCH 01/10] Documentation

In reply to Balbir Singh
Quoted 2 lines · Balbir Singh

* Vivek Goyal <[email protected]> [2009-03-11 21:56:46]:

Thanks for having a look balbir. Sorry for the late reply..

[..]

Quoted 21 lines
Quoted 21 lines
Documentation/block/io-controller.txt new file +21 -0
+Consider following hypothetical scenario. Lets say there are three physical
+disks, namely sda, sdb and sdc. Two logical volumes (lv0 and lv1) have been
+created on top of these. Some part of sdb is in lv0 and some part is in lv1.
+
+			    lv0      lv1
+			  /	\  /     \
+			sda      sdb      sdc
+
+Also consider following cgroup hierarchy
+
+				root
+				/   \
+			       A     B
+			      / \    / \
+			     T1 T2  T3  T4
+
+A and B are two cgroups and T1, T2, T3 and T4 are tasks with-in those cgroups.
+Assuming T1, T2, T3 and T4 are doing IO on lv0 and lv1. These tasks should
+get their fair share of bandwidth on disks sda, sdb and sdc. There is no
+IO control on intermediate logical block nodes (lv0, lv1).
+
Quoted 8 lines

Documentation/block/io-controller.txt line 45

Quoted 5 lines
Documentation/block/io-controller.txt new file +5 -0
+So if tasks T1 and T2 are doing IO on lv0 and T3 and T4 are doing IO on lv1
+only, there will not be any contetion for resources between group A and B if
+IO is going to sda or sdc. But if actual IO gets translated to disk sdb, then
+IO scheduler associated with the sdb will distribute disk bandwidth to
+group A and B proportionate to their weight.

What if we have partitions sda1, sda2 and sda3 instead of sda, sdb and sdc?

As Gui already mentioned, IO control is on per device basis (like IO scheduler) and we don't try to control it per partition basis.

Quoted 15 lines
Quoted 15 lines
Documentation/block/io-controller.txt new file +14 -0
+
+CFQ already has the notion of fairness and it provides differential disk
+access based on priority and class of the task. Just that it is flat and
+with cgroup stuff, it needs to be made hierarchical.
+
+Rest of the IO schedulers (noop, deadline and AS) don't have any notion
+of fairness among various threads.
+
+One of the concerns raised with modifying IO schedulers was that we don't
+want to replicate the code in all the IO schedulers. These patches share
+the fair queuing code which has been moved to a common layer (elevator
+layer). Hence we don't end up replicating code across IO schedulers.
+
+Design
Quoted 8 lines

Documentation/block/io-controller.txt line 64

Quoted 5 lines

+======

Documentation/block/io-controller.txt new file +4 -0
+This patchset primarily uses BFQ (Budget Fair Queuing) code to provide
+fairness among different IO queues. Fabio and Paolo implemented BFQ which uses
+B-WF2Q+ algorithm for fair queuing.
+

References to BFQ, please. I can search them, but having them in the doc would be nice.

That's a good point. In next posting I will put references also.

Quoted 18 lines
Quoted 5 lines
Documentation/block/io-controller.txt new file +4 -0
+Why BFQ?
+
+- Not sure if weighted round robin logic of CFQ can be easily extended for
+  hierarchical mode. One of the things is that we can not keep dividing

Documentation/block/io-controller.txt line 78

Quoted 10 lines
Documentation/block/io-controller.txt new file +10 -0
+  the time slice of parent group among childrens. Deeper we go in hierarchy
+  time slice will get smaller.
+
+  One of the ways to implement hierarchical support could be to keep track
+  of virtual time and service provided to queue/group and select a queue/group
+  for service based on any of the various available algoriths.
+
+  BFQ already had support for hierarchical scheduling, taking those patches
+  was easier.
+

Could you elaborate, when you say timeslices get smaller -

Quoted 2 lines

1. Are you referring to inability to use higher resolution time?
2. Loss of throughput due to timeslice degradation?

I think keeping a track of time using higher resolution time should not be a problem but it would be rather more of loss of throughput due to smaller timeslices and frequent queue switching.

Quoted 10 lines

Documentation/block/io-controller.txt line 85

Quoted 8 lines

+- BFQ was designed to provide tighter bounds/delay w.r.t service provided

Documentation/block/io-controller.txt new file +6 -0
+  to a queue. Delay/Jitter with BFQ is supposed to be O(1).
+
+  Note: BFQ originally used amount of IO done (number of sectors) as notion
+        of service provided. IOW, it tried to provide fairness in terms of
+        actual IO done and not in terms of actual time disk access was
+	given to a queue.

I assume by sectors you mean the kernel sector size?

Yes.

Quoted 19 lines
Quoted 19 lines
Documentation/block/io-controller.txt new file +18 -0
+
+	This patcheset modified BFQ to provide fairness in time domain because
+	that's what CFQ does. So idea was try not to deviate too much from
+	the CFQ behavior initially.
+
+	Providing fairness in time domain makes accounting trciky because
+	due to command queueing, at one time there might be multiple requests
+	from different queues and there is no easy way to find out how much
+	disk time actually was consumed by the requests of a particular
+	queue. More about this in comments in source code.
+
+So it is yet to be seen if changing to time domain still retains BFQ gurantees
+or not.
+
+From data structure point of view, one can think of a tree per device, where
+io groups and io queues are hanging and are being scheduled using B-WF2Q+
+algorithm. io_queue, is end queue where requests are actually stored and
+dispatched from (like cfqq).
Quoted 8 lines

Documentation/block/io-controller.txt line 109

Quoted 6 lines
Documentation/block/io-controller.txt new file +6 -0
+
+These io queues are primarily created by and managed by end io schedulers
+depending on its semantics. For example, noop, deadline and AS ioschedulers
+keep one io queues per cgroup and cfqq keeps one io queue per io_context in
+a cgroup (apart from async queues).
+

I assume there is one io_context per cgroup.

No. There can be multiple io_context per cgroup because currently io_context is defined as threads which are doing IO sharing and are kept in one queue from IO point of view by cfq and multiple queues are not created. So there might be many processes/threads in a cgroup and not necessarily they are sharing the io_context.

Quoted 8 lines

Documentation/block/io-controller.txt line 114

Quoted 6 lines
Documentation/block/io-controller.txt new file +5 -0
+A request is mapped to an io group by elevator layer and which io queue it
+is mapped to with in group depends on ioscheduler. Currently "current" task
+is used to determine the cgroup (hence io group) of the request. Down the
+line we need to make use of bio-cgroup patches to map delayed writes to
+right group.

That seem acceptable

Andrew first wants to see a solid plan for handling async writes :-) So currently I am playing with patches to map writes to correct cgroup. Mapping the IO to right cgroup is only one part of the problem. Other part is that I am not seeing a continuious stream of writes at IO scheduler level. So if two dd processes are running in user space, ideally one can expect two continuous stream of write requests at IO scheduler but instead I see bursty serialized traffic. So a bunch of write request from first dd then another bunch of write requests from second dd and it goes on.. and this leads to no service differentiation between two writes because when higher priority task is not dispatching any IO (for .2 seconds), lower priority task/group gets to use full disk and soon catches up with higher priority one..

Part of this serialization was taking place at request descriptor allocation infrastructure where number of request descriptors are limited and if one writer first consumes most of the descriptors it will block/serialize other writer.

Now I have got a crude working patch where I can limit per group request descriptors so that one group can not block other group. But still don't see continuously backlogged write queues at IO scheduler...

Time to do more debugging and move up the layer and see where this serialization is taking place (i guess page cache...).

Quoted 51 lines
Quoted 51 lines
Documentation/block/io-controller.txt new file +50 -0
+
+Going back to old behavior
+==========================
+In new scheme of things essentially we are creating hierarchical fair
+queuing logic in elevator layer and chaning IO schedulers to make use of
+that logic so that end IO schedulers start supporting hierarchical scheduling.
+
+Elevator layer continues to support the old interfaces. So even if fair queuing
+is enabled at elevator layer, one can have both new hierchical scheduler as
+well as old non-hierarchical scheduler operating.
+
+Also noop, deadline and AS have option of enabling hierarchical scheduling.
+If it is selected, fair queuing is done in hierarchical manner. If hierarchical
+scheduling is disabled, noop, deadline and AS should retain their existing
+behavior.
+
+CFQ is the only exception where one can not disable fair queuing as it is
+needed for provding fairness among various threads even in non-hierarchical
+mode.
+
+Various user visible config options
+===================================
+CONFIG_IOSCHED_NOOP_HIER
+	- Enables hierchical fair queuing in noop. Not selecting this option
+	  leads to old behavior of noop.
+
+CONFIG_IOSCHED_DEADLINE_HIER
+	- Enables hierchical fair queuing in deadline. Not selecting this
+	  option leads to old behavior of deadline.
+
+CONFIG_IOSCHED_AS_HIER
+	- Enables hierchical fair queuing in AS. Not selecting this option
+	  leads to old behavior of AS.
+
+CONFIG_IOSCHED_CFQ_HIER
+	- Enables hierarchical fair queuing in CFQ. Not selecting this option
+	  still does fair queuing among various queus but it is flat and not
+	  hierarchical.
+
+Config options selected automatically
+=====================================
+These config options are not user visible and are selected/deselected
+automatically based on IO scheduler configurations.
+
+CONFIG_ELV_FAIR_QUEUING
+	- Enables/Disables the fair queuing logic at elevator layer.
+
+CONFIG_GROUP_IOSCHED
+	- Enables/Disables hierarchical queuing and associated cgroup bits.
+
Quoted 8 lines

Documentation/block/io-controller.txt line 170

Quoted 6 lines
Documentation/block/io-controller.txt new file +6 -0
+TODO
+====
+- Lots of cleanups, testing, bug fixing, optimizations, benchmarking etc...
+- Convert cgroup ioprio to notion of weight.
+- Anticipatory code will need more work. It is not working properly currently
+  and needs more thought.

What are the problems with the code?

Have not got a chance to look into the issues in detail yet. Just a crude run saw drop in performance. Will debug it later the moment I have got async writes handled...

Quoted 8 lines
Quoted 1 line

+- Use of bio-cgroup patches.

I saw these posted as well

Quoted 2 lines

+- Use of Nauman's per cgroup request descriptor patches. +

More details would be nice, I am not sure I understand

Currently the number of request descriptors which can be allocated per device/request queue are fixed by a sysfs tunable (q->nr_requests). So if there is lots of IO going on from one cgroup then it will consume all the available request descriptors and other cgroup might starve and not get its fair share.

Hence we also need to introduce the notion of request descriptor limit per cgroup so that if request descriptors from one group are exhausted, then it does not impact the IO of other cgroup.

Quoted 15 lines
Quoted 15 lines
Documentation/block/io-controller.txt new file +14 -0
+HOWTO
+=====
+So far I have done very simple testing of running two dd threads in two
+different cgroups. Here is what you can do.
+
+- Enable hierarchical scheduling in io scheuduler of your choice (say cfq).
+	CONFIG_IOSCHED_CFQ_HIER=y
+
+- Compile and boot into kernel and mount IO controller.
+
+	mount -t cgroup -o io none /cgroup
+
+- Create two cgroups
+	mkdir -p /cgroup/test1/ /cgroup/test2
Quoted 8 lines

Documentation/block/io-controller.txt line 192

Quoted 5 lines
Documentation/block/io-controller.txt new file +5 -0
+
+- Set io priority of group test1 and test2
+	echo 0 > /cgroup/test1/io.ioprio
+	echo 4 > /cgroup/test2/io.ioprio
+

What is the meaning of priorities? Which is higher, which is lower? What is the maximum? How does it impact b/w?

Currently cfq has notion of priority range 0-7 (0 being highest). To being with we simply adopted that notion though we are converting it to weights now for group.

Mapping from group priority to group weight is linear. So prio 0 group should get double the BW of prio 4 group.

Thanks
Vivek

A message that is not in the archive (sent off-list or not yet received): [email protected]

#104 lore raw

Re: [PATCH 01/10] Documentation

Quoted 25 lines · Vivek Goyal
Quoted 7 lines

Review

Quoted 6 lines
(quoted diff) +6 -0
+TODO
+====
+- Lots of cleanups, testing, bug fixing, optimizations, benchmarking etc...
+- Convert cgroup ioprio to notion of weight.
+- Anticipatory code will need more work. It is not working properly currently
+  and needs more thought.

What are the problems with the code?

Have not got a chance to look into the issues in detail yet. Just a crude run saw drop in performance. Will debug it later the moment I have got async writes handled...

Quoted 6 lines
Quoted 1 line

+- Use of bio-cgroup patches.

I saw these posted as well

Quoted 2 lines

+- Use of Nauman's per cgroup request descriptor patches. +

More details would be nice, I am not sure I understand

Currently the number of request descriptors which can be allocated per device/request queue are fixed by a sysfs tunable (q->nr_requests). So if there is lots of IO going on from one cgroup then it will consume all the available request descriptors and other cgroup might starve and not get its fair share.

Quoted 3 lines

Hence we also need to introduce the notion of request descriptor limit per cgroup so that if request descriptors from one group are exhausted, then it does not impact the IO of other cgroup.

Unfortunately I couldn't find and I've never seen the Nauman's patches. So I tried to make a patch below against this todo. The reason why I'm posting this despite this is just a quick and ugly hack (and it might be a reinvention of wheel) is that I would like to discuss how we should define the limitation of requests per cgroup.
This patch should be applied on Vivek's I/O controller patches posted on Mar 11.

This patch temporarily distribute q->nr_requests to each cgroup. I think the number should be weighted like BFQ's budget. But in this case, if the hierarchy of cgroup is deep, leaf cgroups are allowed to allocate very few numbers of requests. I don't think this is reasonable...but I don't have specific idea to solve this problem. Does anyone have the good idea?

---
 block/blk-core.c    |   36 +++++++--
 block/blk-sysfs.c   |   22 ++++--
 block/elevator-fq.c |  133 ++++++++++++++++++++++++++++++++--
 block/elevator-fq.h |  201 +++++++++++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 371 insertions(+), 21 deletions(-)
block/blk-core.c +28 -8
index 29bcfac..21023f7 100644
@@ -705,11 +705,15 @@ static void ioc_set_batching(struct request_queue *q, struct io_context *ioc)
 static void __freed_request(struct request_queue *q, int rw)
 {
 	struct request_list *rl = &q->rq;
-
-	if (rl->count[rw] < queue_congestion_off_threshold(q))
+	struct io_group *congested_iog, *full_iog;
+	
+	congested_iog = io_congested_io_group(q, rw);
+	if (rl->count[rw] < queue_congestion_off_threshold(q) &&
+	    !congested_iog)
 		blk_clear_queue_congested(q, rw);
 
-	if (rl->count[rw] + 1 <= q->nr_requests) {
+	full_iog = io_full_io_group(q, rw);
+	if (rl->count[rw] + 1 <= q->nr_requests && !full_iog) {
 		if (waitqueue_active(&rl->wait[rw]))
 			wake_up(&rl->wait[rw]);
 
@@ -721,13 +725,16 @@ static void __freed_request(struct request_queue *q, int rw)
  * A request has just been released.  Account for it, update the full and
  * congestion status, wake up any waiters.   Called under q->queue_lock.
  */
-static void freed_request(struct request_queue *q, int rw, int priv)
+static void freed_request(struct request_queue *q, struct io_group *iog,
+			  int rw, int priv)
 {
 	struct request_list *rl = &q->rq;
 
 	rl->count[rw]--;
 	if (priv)
 		rl->elvpriv--;
+	if (iog)
+		io_group_dec_count(iog, rw);
 
 	__freed_request(q, rw);
 
@@ -746,16 +753,21 @@ static struct request *get_request(struct request_queue *q, int rw_flags,
 {
 	struct request *rq = NULL;
 	struct request_list *rl = &q->rq;
+	struct io_group *iog;
 	struct io_context *ioc = NULL;
 	const int rw = rw_flags & 0x01;
 	int may_queue, priv;
 
+	iog = __io_get_io_group(q);
+
 	may_queue = elv_may_queue(q, rw_flags);
 	if (may_queue == ELV_MQUEUE_NO)
 		goto rq_starved;
 
-	if (rl->count[rw]+1 >= queue_congestion_on_threshold(q)) {
-		if (rl->count[rw]+1 >= q->nr_requests) {
+	if (rl->count[rw]+1 >= queue_congestion_on_threshold(q) ||
+	    io_group_congestion_on(iog, rw)) {
+		if (rl->count[rw]+1 >= q->nr_requests ||
+		    io_group_full(iog, rw)) {
 			ioc = current_io_context(GFP_ATOMIC, q->node);
 			/*
 			 * The queue will fill after this allocation, so set
@@ -789,8 +801,15 @@ static struct request *get_request(struct request_queue *q, int rw_flags,
 	if (rl->count[rw] >= (3 * q->nr_requests / 2))
 		goto out;
 
+	if (iog)
+		if (io_group_count(iog, rw) >=
+		   (3 * io_group_nr_requests(iog) / 2))
+			goto out;
+
 	rl->count[rw]++;
 	rl->starved[rw] = 0;
+	if (iog)
+		io_group_inc_count(iog, rw);
 
 	priv = !test_bit(QUEUE_FLAG_ELVSWITCH, &q->queue_flags);
 	if (priv)
@@ -808,7 +827,7 @@ static struct request *get_request(struct request_queue *q, int rw_flags,
 		 * wait queue, but this is pretty rare.
 		 */
 		spin_lock_irq(q->queue_lock);
-		freed_request(q, rw, priv);
+		freed_request(q, iog, rw, priv);
 
 		/*
 		 * in the very unlikely event that allocation failed and no
@@ -1073,12 +1092,13 @@ void __blk_put_request(struct request_queue *q, struct request *req)
 	if (req->cmd_flags & REQ_ALLOCED) {
 		int rw = rq_data_dir(req);
 		int priv = req->cmd_flags & REQ_ELVPRIV;
+		struct io_group *iog = io_request_io_group(req);
 
 		BUG_ON(!list_empty(&req->queuelist));
 		BUG_ON(!hlist_unhashed(&req->hash));
 
 		blk_free_request(q, req);
-		freed_request(q, rw, priv);
+		freed_request(q, iog, rw, priv);
 	}
 }
 EXPORT_SYMBOL_GPL(__blk_put_request);
block/blk-sysfs.c +14 -8
index 0d98c96..af5191c 100644
@@ -40,6 +40,7 @@ queue_requests_store(struct request_queue *q, const char *page, size_t count)
 {
 	struct request_list *rl = &q->rq;
 	unsigned long nr;
+	int iog_congested[2], iog_full[2];
 	int ret = queue_var_store(&nr, page, count);
 	if (nr < BLKDEV_MIN_RQ)
 		nr = BLKDEV_MIN_RQ;
@@ -47,27 +48,32 @@ queue_requests_store(struct request_queue *q, const char *page, size_t count)
 	spin_lock_irq(q->queue_lock);
 	q->nr_requests = nr;
 	blk_queue_congestion_threshold(q);
+	io_group_set_nrq_all(q, nr, iog_congested, iog_full);
 
-	if (rl->count[READ] >= queue_congestion_on_threshold(q))
+	if (rl->count[READ] >= queue_congestion_on_threshold(q) ||
+	    iog_congested[READ])
 		blk_set_queue_congested(q, READ);
-	else if (rl->count[READ] < queue_congestion_off_threshold(q))
+	else if (rl->count[READ] < queue_congestion_off_threshold(q) &&
+		 !iog_congested[READ])
 		blk_clear_queue_congested(q, READ);
 
-	if (rl->count[WRITE] >= queue_congestion_on_threshold(q))
+	if (rl->count[WRITE] >= queue_congestion_on_threshold(q) ||
+	    iog_congested[WRITE])
 		blk_set_queue_congested(q, WRITE);
-	else if (rl->count[WRITE] < queue_congestion_off_threshold(q))
+	else if (rl->count[WRITE] < queue_congestion_off_threshold(q) &&
+		 !iog_congested[WRITE])
 		blk_clear_queue_congested(q, WRITE);
 
-	if (rl->count[READ] >= q->nr_requests) {
+	if (rl->count[READ] >= q->nr_requests || iog_full[READ]) {
 		blk_set_queue_full(q, READ);
-	} else if (rl->count[READ]+1 <= q->nr_requests) {
+	} else if (rl->count[READ]+1 <= q->nr_requests && !iog_full[READ]) {
 		blk_clear_queue_full(q, READ);
 		wake_up(&rl->wait[READ]);
 	}
 
-	if (rl->count[WRITE] >= q->nr_requests) {
+	if (rl->count[WRITE] >= q->nr_requests || iog_full[WRITE]) {
 		blk_set_queue_full(q, WRITE);
-	} else if (rl->count[WRITE]+1 <= q->nr_requests) {
+	} else if (rl->count[WRITE]+1 <= q->nr_requests && !iog_full[WRITE]) {
 		blk_clear_queue_full(q, WRITE);
 		wake_up(&rl->wait[WRITE]);
 	}
block/elevator-fq.c +128 -5

180 lines not on this page · raw message

block/elevator-fq.h +201 -0

242 lines not on this page · raw message

Signature · 7 lines
1.5.4.3


-- 
IKEDA, Munehiro
 NEC Corporation of America
   m-ikeda-MDRzhb/z0dd8UrSeD/[email protected]
#105 lore raw

Re: [PATCH 01/10] Documentation

In reply to Vivek Goyal
Quoted 25 lines · Vivek Goyal
Quoted 7 lines

Review

Quoted 6 lines
(quoted diff) +6 -0
+TODO
+====
+- Lots of cleanups, testing, bug fixing, optimizations, benchmarking etc...
+- Convert cgroup ioprio to notion of weight.
+- Anticipatory code will need more work. It is not working properly currently
+  and needs more thought.

What are the problems with the code?

Have not got a chance to look into the issues in detail yet. Just a crude run saw drop in performance. Will debug it later the moment I have got async writes handled...

Quoted 6 lines
Quoted 1 line

+- Use of bio-cgroup patches.

I saw these posted as well

Quoted 2 lines

+- Use of Nauman's per cgroup request descriptor patches. +

More details would be nice, I am not sure I understand

Currently the number of request descriptors which can be allocated per device/request queue are fixed by a sysfs tunable (q->nr_requests). So if there is lots of IO going on from one cgroup then it will consume all the available request descriptors and other cgroup might starve and not get its fair share.

Quoted 3 lines

Hence we also need to introduce the notion of request descriptor limit per cgroup so that if request descriptors from one group are exhausted, then it does not impact the IO of other cgroup.

Unfortunately I couldn't find and I've never seen the Nauman's patches. So I tried to make a patch below against this todo. The reason why I'm posting this despite this is just a quick and ugly hack (and it might be a reinvention of wheel) is that I would like to discuss how we should define the limitation of requests per cgroup.
This patch should be applied on Vivek's I/O controller patches posted on Mar 11.

This patch temporarily distribute q->nr_requests to each cgroup. I think the number should be weighted like BFQ's budget. But in this case, if the hierarchy of cgroup is deep, leaf cgroups are allowed to allocate very few numbers of requests. I don't think this is reasonable...but I don't have specific idea to solve this problem. Does anyone have the good idea?

---

block/blk-core.c    |   36 +++++++--
block/blk-sysfs.c   |   22 ++++--
block/elevator-fq.c |  133 ++++++++++++++++++++++++++++++++--
block/elevator-fq.h |  201 +++++++++++++++++++++++++++++++++++++++++++++++++++
4 files changed, 371 insertions(+), 21 deletions(-)
block/blk-core.c +0 -0
index 29bcfac..21023f7 100644
@@ -705,11 +705,15 @@ static void ioc_set_batching(struct request_queue *q, struct io_context *ioc)

static void __freed_request(struct request_queue *q, int rw)

{
	struct request_list *rl = &q->rq;

-
- if (rl->count[rw] < queue_congestion_off_threshold(q))
+ struct io_group *congested_iog, *full_iog;
+
+ congested_iog = io_congested_io_group(q, rw);
+ if (rl->count[rw] < queue_congestion_off_threshold(q) &&
+ !congested_iog)

		blk_clear_queue_congested(q, rw);

- if (rl->count[rw] + 1 <= q->nr_requests) {
+ full_iog = io_full_io_group(q, rw);

+	if (rl->count[rw] + 1 <= q->nr_requests && !full_iog) {
		if (waitqueue_active(&rl->wait[rw]))
			wake_up(&rl->wait[rw]);

@@ -721,13 +725,16 @@ static void __freed_request(struct request_queue *q, int rw)
* A request has just been released. Account for it, update the full and
* congestion status, wake up any waiters. Called under q->queue_lock.
*/
-static void freed_request(struct request_queue *q, int rw, int priv) +static void freed_request(struct request_queue *q, struct io_group *iog,
+ int rw, int priv)

{
	struct request_list *rl = &q->rq;

	rl->count[rw]--;
	if (priv)
		rl->elvpriv--;

+ if (iog)
+ io_group_dec_count(iog, rw);

	__freed_request(q, rw);

@@ -746,16 +753,21 @@ static struct request *get_request(struct request_queue *q, int rw_flags,

{
	struct request *rq = NULL;
	struct request_list *rl = &q->rq;
+	struct io_group *iog;
	struct io_context *ioc = NULL;
	const int rw = rw_flags & 0x01;
	int may_queue, priv;

+ iog = __io_get_io_group(q);
+

	may_queue = elv_may_queue(q, rw_flags);
	if (may_queue == ELV_MQUEUE_NO)
		goto rq_starved;

- if (rl->count[rw]+1 >= queue_congestion_on_threshold(q)) {
- if (rl->count[rw]+1 >= q->nr_requests) {
+ if (rl->count[rw]+1 >= queue_congestion_on_threshold(q) ||
+ io_group_congestion_on(iog, rw)) {
+ if (rl->count[rw]+1 >= q->nr_requests ||

+		    io_group_full(iog, rw)) {
			ioc = current_io_context(GFP_ATOMIC, q->node);
			/*
			 * The queue will fill after this allocation, so set
@@ -789,8 +801,15 @@ static struct request *get_request(struct request_queue *q, int rw_flags,
	if (rl->count[rw] >= (3 * q->nr_requests / 2))
		goto out;

+ if (iog)
+ if (io_group_count(iog, rw) >=
+ (3 * io_group_nr_requests(iog) / 2))
+ goto out;
+

	rl->count[rw]++;
	rl->starved[rw] = 0;

+ if (iog)
+ io_group_inc_count(iog, rw);

	priv = !test_bit(QUEUE_FLAG_ELVSWITCH, &q->queue_flags);
	if (priv)
@@ -808,7 +827,7 @@ static struct request *get_request(struct request_queue *q, int rw_flags,
		 * wait queue, but this is pretty rare.
		 */
		spin_lock_irq(q->queue_lock);

- freed_request(q, rw, priv);
+ freed_request(q, iog, rw, priv);

		/*
		 * in the very unlikely event that allocation failed and no
@@ -1073,12 +1092,13 @@ void __blk_put_request(struct request_queue *q, struct request *req)
	if (req->cmd_flags & REQ_ALLOCED) {
		int rw = rq_data_dir(req);
		int priv = req->cmd_flags & REQ_ELVPRIV;

+ struct io_group *iog = io_request_io_group(req);

		BUG_ON(!list_empty(&req->queuelist));
		BUG_ON(!hlist_unhashed(&req->hash));

		blk_free_request(q, req);
-		freed_request(q, rw, priv);
+		freed_request(q, iog, rw, priv);
	}
}

EXPORT_SYMBOL_GPL(__blk_put_request);

block/blk-sysfs.c +0 -0
index 0d98c96..af5191c 100644
@@ -40,6 +40,7 @@ queue_requests_store(struct request_queue *q, const char *page, size_t count)
{
	struct request_list *rl = &q->rq;
	unsigned long nr;
+	int iog_congested[2], iog_full[2];
	int ret = queue_var_store(&nr, page, count);
	if (nr < BLKDEV_MIN_RQ)
		nr = BLKDEV_MIN_RQ;
@@ -47,27 +48,32 @@ queue_requests_store(struct request_queue *q, const char *page, size_t count)
	spin_lock_irq(q->queue_lock);
	q->nr_requests = nr;
	blk_queue_congestion_threshold(q);

+ io_group_set_nrq_all(q, nr, iog_congested, iog_full);

- if (rl->count[READ] >= queue_congestion_on_threshold(q))
+ if (rl->count[READ] >= queue_congestion_on_threshold(q) ||
+ iog_congested[READ])

		blk_set_queue_congested(q, READ);
-	else if (rl->count[READ] < queue_congestion_off_threshold(q))
+	else if (rl->count[READ] < queue_congestion_off_threshold(q) &&
+		 !iog_congested[READ])
		blk_clear_queue_congested(q, READ);

- if (rl->count[WRITE] >= queue_congestion_on_threshold(q))
+ if (rl->count[WRITE] >= queue_congestion_on_threshold(q) ||
+ iog_congested[WRITE])

		blk_set_queue_congested(q, WRITE);
-	else if (rl->count[WRITE] < queue_congestion_off_threshold(q))
+	else if (rl->count[WRITE] < queue_congestion_off_threshold(q) &&
+		 !iog_congested[WRITE])
		blk_clear_queue_congested(q, WRITE);

- if (rl->count[READ] >= q->nr_requests) {

+	if (rl->count[READ] >= q->nr_requests || iog_full[READ]) {
		blk_set_queue_full(q, READ);
-	} else if (rl->count[READ]+1 <= q->nr_requests) {
+	} else if (rl->count[READ]+1 <= q->nr_requests && !iog_full[READ]) {
		blk_clear_queue_full(q, READ);
		wake_up(&rl->wait[READ]);
	}

- if (rl->count[WRITE] >= q->nr_requests) {

+	if (rl->count[WRITE] >= q->nr_requests || iog_full[WRITE]) {
		blk_set_queue_full(q, WRITE);
-	} else if (rl->count[WRITE]+1 <= q->nr_requests) {
+	} else if (rl->count[WRITE]+1 <= q->nr_requests && !iog_full[WRITE]) {
		blk_clear_queue_full(q, WRITE);
		wake_up(&rl->wait[WRITE]);
	}
block/elevator-fq.c +0 -0
index df53418..3b021f3 100644
@@ -924,6 +924,111 @@ struct io_group *io_lookup_io_group_current(struct request_queue *q)

}
EXPORT_SYMBOL(io_lookup_io_group_current);

+/*
+ * TODO
+ * This is complete dupulication of blk_queue_congestion_threshold()
+ * except for the argument type and name. Can we merge them?
+ */
+static void io_group_nrq_congestion_threshold(struct io_group_nrq *nrq) +{
+ int nr;
+
+ nr = nrq->nr_requests - (nrq->nr_requests / 8) + 1;
+ if (nr > nrq->nr_requests)
+ nr = nrq->nr_requests;
+ nrq->nr_congestion_on = nr;
+
+ nr = nrq->nr_requests - (nrq->nr_requests / 8)
+ - (nrq->nr_requests / 16) - 1;
+ if (nr < 1)
+ nr = 1;
+ nrq->nr_congestion_off = nr;
+}
+
+static void io_group_set_nrq(struct io_group_nrq *nrq, int nr_requests,
+ int *congested, int *full)
+{
+ int i;
+
+ BUG_ON(nr_requests < 0);
+
+ nrq->nr_requests = nr_requests;
+ io_group_nrq_congestion_threshold(nrq);
+
+ for (i=0; i<2; i++) {
+ if (nrq->count[i] >= nrq->nr_congestion_on)
+ congested[i] = 1;
+ else if (nrq->count[i] < nrq->nr_congestion_off)
+ congested[i] = 0;
+
+ if (nrq->count[i] >= nrq->nr_requests)
+ full[i] = 1;
+ else if (nrq->count[i]+1 <= nrq->nr_requests)
+ full[i] = 0;
+ }
+}
+
+void io_group_set_nrq_all(struct request_queue *q, int nr,
+ int *congested, int *full)
+{
+ struct elv_fq_data *efqd = &q->elevator->efqd;
+ struct hlist_head *head = &efqd->group_list;
+ struct io_group *root = efqd->root_group;
+ struct hlist_node *n;
+ struct io_group *iog;
+ struct io_group_nrq *nrq;
+ int nrq_congested[2];
+ int nrq_full[2];
+ int i;
+
+ for (i=0; i<2; i++)
+ *(congested + i) = *(full + i) = 0;
+
+ nrq = &root->nrq;
+ io_group_set_nrq(nrq, nr, nrq_congested, nrq_full);
+ for (i=0; i<2; i++) {
+ *(congested + i) |= nrq_congested[i];
+ *(full + i) |= nrq_full[i];
+ }
+
+ hlist_for_each_entry(iog, n, head, elv_data_node) {
+ nrq = &iog->nrq;
+ io_group_set_nrq(nrq, nr, nrq_congested, nrq_full);
+ for (i=0; i<2; i++) {
+ *(congested + i) |= nrq_congested[i];
+ *(full + i) |= nrq_full[i];
+ }
+ }
+}
+
+struct io_group *io_congested_io_group(struct request_queue *q, int rw) +{
+ struct hlist_head *head = &q->elevator->efqd.group_list;
+ struct hlist_node *n;
+ struct io_group *iog;
+
+ hlist_for_each_entry(iog, n, head, elv_data_node) {
+ struct io_group_nrq *nrq = &iog->nrq;
+ if (nrq->count[rw] >= nrq->nr_congestion_off)
+ return iog;
+ }
+ return NULL;
+}
+
+struct io_group *io_full_io_group(struct request_queue *q, int rw) +{
+ struct hlist_head *head = &q->elevator->efqd.group_list;
+ struct hlist_node *n;
+ struct io_group *iog;
+
+ hlist_for_each_entry(iog, n, head, elv_data_node) {
+ struct io_group_nrq *nrq = &iog->nrq;
+ if (nrq->count[rw] >= nrq->nr_requests)
+ return iog;
+ }
+ return NULL;
+}
+
void io_group_init_entity(struct io_cgroup *iocg, struct io_group *iog)

{
	struct io_entity *entity = &iog->entity;
@@ -934,6 +1039,12 @@ void io_group_init_entity(struct io_cgroup *iocg, struct io_group *iog)
	entity->my_sched_data = &iog->sched_data;
}

+static void io_group_init_nrq(struct request_queue *q, struct io_group_nrq *nrq) +{
+ nrq->nr_requests = q->nr_requests;
+ io_group_nrq_congestion_threshold(nrq);
+}
+
void io_group_set_parent(struct io_group *iog, struct io_group *parent)

{
	struct io_entity *entity;
@@ -1053,6 +1164,8 @@ struct io_group *io_group_chain_alloc(struct request_queue *q, void *key,
		io_group_init_entity(iocg, iog);
		iog->my_entity = &iog->entity;

+ io_group_init_nrq(q, &iog->nrq);
+

		if (leaf == NULL) {
			leaf = iog;
			prev = leaf;

@@ -1176,7 +1289,7 @@ struct io_group *io_find_alloc_group(struct request_queue *q,
* Generic function to make sure cgroup hierarchy is all setup once a request
* from a cgroup is received by the io scheduler.
*/
-struct io_group *io_get_io_group(struct request_queue *q)
+struct io_group *__io_get_io_group(struct request_queue *q)

{
	struct cgroup *cgroup;
	struct io_group *iog;
@@ -1192,6 +1305,19 @@ struct io_group *io_get_io_group(struct request_queue *q)
	return iog;
}

+struct io_group *io_get_io_group(struct request_queue *q) +{
+ struct io_group *iog;
+ unsigned long flags;
+
+ spin_lock_irqsave(q->queue_lock, flags);
+ iog = __io_get_io_group(q);
+ spin_unlock_irqrestore(q->queue_lock, flags);
+ BUG_ON(!iog);
+
+ return iog;
+}
+
void io_free_root_group(struct elevator_queue *e)

{
	struct io_cgroup *iocg = &io_root_cgroup;
@@ -1220,6 +1346,7 @@ struct io_group *io_alloc_root_group(struct request_queue *q,
	iog->entity.parent = NULL;
	for (i = 0; i < IO_IOPRIO_CLASSES; i++)
		iog->sched_data.service_tree[i] = IO_SERVICE_TREE_INIT;

+ io_group_init_nrq(q, &iog->nrq);

	iocg = &io_root_cgroup;
	spin_lock_irq(&iocg->lock);
@@ -1533,15 +1660,11 @@ void elv_fq_set_request_io_group(struct request_queue *q,
						struct request *rq)
{
	struct io_group *iog;

- unsigned long flags;

/* Make sure io group hierarchy has been setup and also set the
* io group to which rq belongs. Later we should make use of
* bio cgroup patches to determine the io group */

-	spin_lock_irqsave(q->queue_lock, flags);
	iog = io_get_io_group(q);

- spin_unlock_irqrestore(q->queue_lock, flags);
- BUG_ON(!iog);

	/* Store iog in rq. TODO: take care of referencing */
	rq->iog = iog;
block/elevator-fq.h +0 -0
index fc4110d..f8eabd4 100644
@@ -187,6 +187,22 @@ struct io_queue {
 

#ifdef CONFIG_GROUP_IOSCHED
/**
+ * struct io_group_nrq - structure to store allocated requests info
+ * @nr_requests: maximun num of requests for the io_group
+ * @nr_congestion_on: threshold to determin the io_group is cogested.
+ * @nr_congestion_off: threshold to determin the io_group is not congested.
+ * @count: num of allocated requests.
+ *
+ * All fields are protected by queue_lock.
+ */
+struct io_group_nrq {
+ unsigned long nr_requests;
+ unsigned int nr_congestion_on;
+ unsigned int nr_congestion_off;
+ int count[2];
+};
+
+/**
* struct bfq_group - per (device, cgroup) data structure.
* @entity: schedulable entity to insert into the parent group sched_data.
* @sched_data: own sched_data, to contain child entities (they may be
@@ -235,6 +251,8 @@ struct io_group {

	/* Single ioq per group, used for noop, deadline, anticipatory */
	struct io_queue *ioq;

+
+ struct io_group_nrq nrq;
};

/**
@@ -469,6 +487,11 @@ extern int elv_fq_set_request_ioq(struct request_queue *q, struct request *rq, extern void elv_fq_unset_request_ioq(struct request_queue *q,

					struct request *rq);

extern struct io_queue *elv_lookup_ioq_current(struct request_queue *q); +extern void io_group_set_nrq_all(struct request_queue *q, int nr,
+ int *congested, int *full);
+extern struct io_group *io_congested_io_group(struct request_queue *q, int rw); +extern struct io_group *io_full_io_group(struct request_queue *q, int rw); +extern struct io_group *__io_get_io_group(struct request_queue *q);

/* Returns single ioq associated with the io group. */
static inline struct io_queue *io_group_ioq(struct io_group *iog)
@@ -486,6 +509,52 @@ static inline void io_group_set_ioq(struct io_group *iog, struct io_queue *ioq)

	iog->ioq = ioq;
}

+static inline struct io_group *io_request_io_group(struct request *rq)
+{
+ return rq->iog;
+}
+
+static inline unsigned long io_group_nr_requests(struct io_group *iog)
+{
+ BUG_ON(!iog);
+ return iog->nrq.nr_requests;
+}
+
+static inline int io_group_inc_count(struct io_group *iog, int rw)
+{
+ BUG_ON(!iog);
+ return iog->nrq.count[rw]++;
+}
+
+static inline int io_group_dec_count(struct io_group *iog, int rw)
+{
+ BUG_ON(!iog);
+ return iog->nrq.count[rw]--;
+}
+
+static inline int io_group_count(struct io_group *iog, int rw)
+{
+ BUG_ON(!iog);
+ return iog->nrq.count[rw];
+}
+
+static inline int io_group_congestion_on(struct io_group *iog, int rw)
+{
+ BUG_ON(!iog);
+ return iog->nrq.count[rw] + 1 >= iog->nrq.nr_congestion_on;
+}
+
+static inline int io_group_congestion_off(struct io_group *iog, int rw)
+{
+ BUG_ON(!iog);
+ return iog->nrq.count[rw] < iog->nrq.nr_congestion_off;
+}
+
+static inline int io_group_full(struct io_group *iog, int rw)
+{
+ BUG_ON(!iog);
+ return iog->nrq.count[rw] + 1 >= iog->nrq.nr_requests;
+}
#else /* !GROUP_IOSCHED */
/*
* No ioq movement is needed in case of flat setup. root io group gets cleaned
@@ -537,6 +606,71 @@ static inline struct io_queue *elv_lookup_ioq_current(struct request_queue *q)

	return NULL;
}

+static inline void io_group_set_nrq_all(struct request_queue *q, int nr,
+ int *congested, int *full)
+{
+ int i;
+ for (i=0; i<2; i++)
+ *(congested + i) = *(full + i) = 0;
+}
+
+static inline struct io_group *
+io_congested_io_group(struct request_queue *q, int rw)
+{
+ return NULL;
+}
+
+static inline struct io_group *
+io_full_io_group(struct request_queue *q, int rw)
+{
+ return NULL;
+}
+
+static inline struct io_group *__io_get_io_group(struct request_queue *q) +{
+ return NULL;
+}
+
+static inline struct io_group *io_request_io_group(struct request *rq) +{
+ return NULL;
+}
+
+static inline unsigned long io_group_nr_requests(struct io_group *iog) +{
+ return 0;
+}
+
+static inline int io_group_inc_count(struct io_group *iog, int rw) +{
+ return 0;
+}
+
+static inline int io_group_dec_count(struct io_group *iog, int rw) +{
+ return 0;
+}
+
+static inline int io_group_count(struct io_group *iog, int rw) +{
+ return 0;
+}
+
+static inline int io_group_congestion_on(struct io_group *iog, int rw) +{
+ return 0;
+}
+
+static inline int io_group_congestion_off(struct io_group *iog, int rw) +{
+ return 1;
+}
+
+static inline int io_group_full(struct io_group *iog, int rw) +{
+ return 0;
+}
+
#endif /* GROUP_IOSCHED */

/* Functions used by blksysfs.c */
@@ -589,6 +723,9 @@ extern void elv_free_ioq(struct io_queue *ioq);

#else /* CONFIG_ELV_FAIR_QUEUING */

+struct io_group {
+};
+
static inline int elv_init_fq_data(struct request_queue *q,
struct elevator_queue *e)
{
@@ -655,5 +792,69 @@ static inline struct io_queue *elv_lookup_ioq_current(struct request_queue *q)

	return NULL;
}

+static inline void io_group_set_nrq_all(struct request_queue *q, int nr,
+ int *congested, int *full)
+{
+ int i;
+ for (i=0; i<2; i++)
+ *(congested + i) = *(full + i) = 0;
+}
+
+static inline struct io_group *
+io_congested_io_group(struct request_queue *q, int rw)
+{
+ return NULL;
+}
+
+static inline struct io_group *
+io_full_io_group(struct request_queue *q, int rw)
+{
+ return NULL;
+}
+
+static inline struct io_group *__io_get_io_group(struct request_queue *q) +{
+ return NULL;
+}
+
+static inline struct io_group *io_request_io_group(struct request *rq) +{
+ return NULL;
+}
+
+static inline unsigned long io_group_nr_requests(struct io_group *iog) +{
+ return 0;
+}
+
+static inline int io_group_inc_count(struct io_group *iog, int rw) +{
+ return 0;
+}
+
+static inline int io_group_dec_count(struct io_group *iog, int rw) +{
+ return 0;
+}
+
+static inline int io_group_count(struct io_group *iog, int rw) +{
+ return 0;
+}
+
+static inline int io_group_congestion_on(struct io_group *iog, int rw) +{
+ return 0;
+}
+
+static inline int io_group_congestion_off(struct io_group *iog, int rw) +{
+ return 1;
+}
+
+static inline int io_group_full(struct io_group *iog, int rw) +{
+ return 0;
+}
#endif /* CONFIG_ELV_FAIR_QUEUING */
#endif /* _BFQ_SCHED_H */

Signature · 7 lines
1.5.4.3


-- 
IKEDA, Munehiro
NEC Corporation of America
  [email protected]
#106 lore raw

IO Controller per cgroup request descriptors (Re: [PATCH 01/10] Documentation)

Quoted 30 lines · Munehiro
Quoted 25 lines · Vivek Goyal
Quoted 7 lines

Review

Quoted 6 lines
(quoted diff) +6 -0
+TODO
+====
+- Lots of cleanups, testing, bug fixing, optimizations, benchmarking etc...
+- Convert cgroup ioprio to notion of weight.
+- Anticipatory code will need more work. It is not working properly currently
+  and needs more thought.

What are the problems with the code?

Have not got a chance to look into the issues in detail yet. Just a crude run saw drop in performance. Will debug it later the moment I have got async writes handled...

Quoted 6 lines
Quoted 1 line

+- Use of bio-cgroup patches.

I saw these posted as well

Quoted 2 lines

+- Use of Nauman's per cgroup request descriptor patches. +

More details would be nice, I am not sure I understand

Currently the number of request descriptors which can be allocated per device/request queue are fixed by a sysfs tunable (q->nr_requests). So if there is lots of IO going on from one cgroup then it will consume all the available request descriptors and other cgroup might starve and not get its fair share.

Quoted 3 lines

Hence we also need to introduce the notion of request descriptor limit per cgroup so that if request descriptors from one group are exhausted, then it does not impact the IO of other cgroup.

Quoted 7 lines

Unfortunately I couldn't find and I've never seen the Nauman's patches. So I tried to make a patch below against this todo. The reason why I'm posting this despite this is just a quick and ugly hack (and it might be a reinvention of wheel) is that I would like to discuss how we should define the limitation of requests per cgroup.
This patch should be applied on Vivek's I/O controller patches posted on Mar 11.

Hi IKEDA,

Sorry for the confusion here. Actually Nauman had sent a patch to select group of people who were initially copied on the mail thread.

Quoted 8 lines

This patch temporarily distribute q->nr_requests to each cgroup. I think the number should be weighted like BFQ's budget. But in this case, if the hierarchy of cgroup is deep, leaf cgroups are allowed to allocate very few numbers of requests. I don't think this is reasonable...but I don't have specific idea to solve this problem. Does anyone have the good idea?

Thanks for the patch. Yes, ideally one would expect the request descriptor to be allocated also in proportion to the weight but I guess that would become very comlicated.

In terms of simpler things, two thoughts come to mind.

- First approach is to make q->nr_requests per group. So every group is entitled for q->nr_requests as set by the user. This is what your patch seems to have done.

I had some concerns with this approach. First of all it does not seem to have an upper bound on number of request descriptors allocated per queue because if a user creates more cgroups, total number of request descriptors increase.

- Second approach can be that we retain the meaning of q->nr_requests which defines the total number of request descriptors on the queue (with the exception of 50% more descriptors for batching processes). And we define a new per group limit q->nr_group_requests which defines how many requests per group can be assigned. So q->nr_requests defines total pool size on the queue and q->nr_group_requests will define how many requests each group can allocate out of that pool.

Here the issue is that a user shall have to balance the q->nr_group_requests and q->nr_requests properly.

To experiment, I have implemented the second approach. I am attaching the patch which is in my current tree. It probably will not apply on my march 11 posting as since then patches have changed. But posting it here so that at least it will give an idea behind the thought process.

Ideas are welcome...

Thanks
Vivek

o Currently a request queue has got fixed number of request descriptors for sync and async requests. Once the request descriptors are consumed, new processes are put to sleep and they effectively become serialized. Because sync and async queues are separate, async requests don't impact sync ones but if one is looking for fairness between async requests, that is not achievable if request queue descriptors become bottleneck.

o Make request descriptor's per io group so that if there is lots of IO going on in one cgroup, it does not impact the IO of other group.

o This patch implements the per cgroup request descriptors. request pool per queue is still common but every group will have its own wait list and its own count of request descriptors allocated to that group for sync and async queues. So effectively request_list becomes per io group property and not a global request queue feature.

o Currently one can define q->nr_requests to limit request descriptors allocated for the queue. Now there is another tunable q->nr_group_requests which controls the requests descriptr limit per group. q->nr_requests supercedes q->nr_group_requests to make sure if there are lots of groups present, we don't end up allocating too many request descriptors on the queue.

o Issues: Currently notion of congestion is per queue. With per group request descriptor it is possible that queue is not congested but the group bio will go into is congested.

---
 block/blk-core.c       |  216 ++++++++++++++++++++++++++++++++++---------------
 block/blk-settings.c   |    3 
 block/blk-sysfs.c      |   57 ++++++++++--
 block/elevator-fq.c    |   15 +++
 block/elevator-fq.h    |    8 +
 block/elevator.c       |    6 -
 include/linux/blkdev.h |   62 +++++++++++++-
 7 files changed, 287 insertions(+), 80 deletions(-)
include/linux/blkdev.h +60 -2
Index: linux9/include/linux/blkdev.h
===================================================================
@@ -32,21 +32,51 @@ struct request;
 struct sg_io_hdr;
 
 #define BLKDEV_MIN_RQ	4
+
+#ifdef CONFIG_GROUP_IOSCHED
+#define BLKDEV_MAX_RQ	256	/* Default maximum */
+#define BLKDEV_MAX_GROUP_RQ    64      /* Default maximum */
+#else
 #define BLKDEV_MAX_RQ	128	/* Default maximum */
+/*
+ * This is eqivalent to case of only one group present (root group). Let
+ * it consume all the request descriptors available on the queue .
+ */
+#define BLKDEV_MAX_GROUP_RQ    BLKDEV_MAX_RQ      /* Default maximum */
+#endif
 
 struct request;
 typedef void (rq_end_io_fn)(struct request *, int);
 
 struct request_list {
 	/*
-	 * count[], starved[], and wait[] are indexed by
+	 * count[], starved and wait[] are indexed by
 	 * BLK_RW_SYNC/BLK_RW_ASYNC
 	 */
 	int count[2];
 	int starved[2];
+	wait_queue_head_t wait[2];
+};
+
+/*
+ * This data structures keeps track of mempool of requests for the queue
+ * and some overall statistics.
+ */
+struct request_data {
+	/*
+	 * Per queue request descriptor count. This is in addition to per
+	 * cgroup count
+	 */
+	int count[2];
 	int elvpriv;
 	mempool_t *rq_pool;
-	wait_queue_head_t wait[2];
+	int starved;
+	/*
+	 * Global list for starved tasks. A task will be queued here if
+	 * it could not allocate request descriptor and the associated
+	 * group request list does not have any requests pending.
+	 */
+	wait_queue_head_t starved_wait;
 };
 
 /*
@@ -251,6 +281,7 @@ struct request {
 #ifdef CONFIG_GROUP_IOSCHED
 	/* io group request belongs to */
 	struct io_group *iog;
+	struct request_list *rl;
 #endif /* GROUP_IOSCHED */
 #endif /* ELV_FAIR_QUEUING */
 };
@@ -340,6 +371,9 @@ struct request_queue
 	 */
 	struct request_list	rq;
 
+	/* Contains request pool and other data like starved data */
+	struct request_data	rq_data;
+
 	request_fn_proc		*request_fn;
 	make_request_fn		*make_request_fn;
 	prep_rq_fn		*prep_rq_fn;
@@ -402,6 +436,8 @@ struct request_queue
 	 * queue settings
 	 */
 	unsigned long		nr_requests;	/* Max # of requests */
+	/* Max # of per io group requests */
+	unsigned long		nr_group_requests;
 	unsigned int		nr_congestion_on;
 	unsigned int		nr_congestion_off;
 	unsigned int		nr_batching;
@@ -773,6 +809,28 @@ extern int scsi_cmd_ioctl(struct request
 extern int sg_scsi_ioctl(struct request_queue *, struct gendisk *, fmode_t,
 			 struct scsi_ioctl_command __user *);
 
+extern void blk_init_request_list(struct request_list *rl);
+
+static inline struct request_list *blk_get_request_list(struct request_queue *q,
+						struct bio *bio)
+{
+#ifdef CONFIG_GROUP_IOSCHED
+	return io_group_get_request_list(q, bio);
+#else
+	return &q->rq;
+#endif
+}
+
+static inline struct request_list *rq_rl(struct request_queue *q,
+						struct request *rq)
+{
+#ifdef CONFIG_GROUP_IOSCHED
+	return rq->rl;
+#else
+	return blk_get_request_list(q, NULL);
+#endif
+}
+
 /*
  * Temporary export, until SCSI gets fixed up.
  */
block/elevator.c +3 -3
Index: linux9/block/elevator.c
===================================================================
@@ -664,7 +664,7 @@ void elv_quiesce_start(struct request_qu
 	 * make sure we don't have any requests in flight
 	 */
 	elv_drain_elevator(q);
-	while (q->rq.elvpriv) {
+	while (q->rq_data.elvpriv) {
 		blk_start_queueing(q);
 		spin_unlock_irq(q->queue_lock);
 		msleep(10);
@@ -764,8 +764,8 @@ void elv_insert(struct request_queue *q,
 	}
 
 	if (unplug_it && blk_queue_plugged(q)) {
-		int nrq = q->rq.count[BLK_RW_SYNC] + q->rq.count[BLK_RW_ASYNC]
-			- q->in_flight;
+		int nrq = q->rq_data.count[BLK_RW_SYNC] +
+				q->rq_data.count[BLK_RW_ASYNC] - q->in_flight;
 
 		if (nrq >= q->unplug_thresh)
 			__generic_unplug_device(q);
block/blk-core.c +153 -63

360 lines not on this page · raw message

block/blk-sysfs.c +45 -12

108 lines not on this page · raw message

block/blk-settings.c +3 -0
Index: linux9/block/blk-settings.c
===================================================================
@@ -123,6 +123,9 @@ void blk_queue_make_request(struct reque
 	 * set defaults
 	 */
 	q->nr_requests = BLKDEV_MAX_RQ;
+#ifdef CONFIG_GROUP_IOSCHED
+	q->nr_group_requests = BLKDEV_MAX_GROUP_RQ;
+#endif
 	blk_queue_max_phys_segments(q, MAX_PHYS_SEGMENTS);
 	blk_queue_max_hw_segments(q, MAX_HW_SEGMENTS);
 	blk_queue_segment_boundary(q, BLK_SEG_BOUNDARY_MASK);
block/elevator-fq.c +15 -0
Index: linux9/block/elevator-fq.c
===================================================================
@@ -954,6 +954,17 @@ struct io_cgroup *cgroup_to_io_cgroup(st
 			    struct io_cgroup, css);
 }
 
+struct request_list *io_group_get_request_list(struct request_queue *q,
+						struct bio *bio)
+{
+	struct io_group *iog;
+
+	iog = io_get_io_group_bio(q, bio, 1);
+	BUG_ON(!iog);
+out:
+	return &iog->rl;
+}
+
 /*
  * Search the bfq_group for bfqd into the hash table (by now only a list)
  * of bgrp.  Must be called under rcu_read_lock().
@@ -1203,6 +1214,8 @@ struct io_group *io_group_chain_alloc(st
 		io_group_init_entity(iocg, iog);
 		iog->my_entity = &iog->entity;
 
+		blk_init_request_list(&iog->rl);
+
 		if (leaf == NULL) {
 			leaf = iog;
 			prev = leaf;
@@ -1446,6 +1459,8 @@ struct io_group *io_alloc_root_group(str
 	for (i = 0; i < IO_IOPRIO_CLASSES; i++)
 		iog->sched_data.service_tree[i] = IO_SERVICE_TREE_INIT;
 
+	blk_init_request_list(&iog->rl);
+
 	iocg = &io_root_cgroup;
 	spin_lock_irq(&iocg->lock);
 	rcu_assign_pointer(iog->key, key);
block/elevator-fq.h +8 -0
Index: linux9/block/elevator-fq.h
===================================================================
@@ -239,8 +239,14 @@ struct io_group {
 
 	/* Single ioq per group, used for noop, deadline, anticipatory */
 	struct io_queue *ioq;
+
+	/* request list associated with the group */
+	struct request_list rl;
 };
 
+#define IOG_FLAG_READFULL	1	/* read queue has been filled */
+#define IOG_FLAG_WRITEFULL	2	/* write queue has been filled */
+
 /**
  * struct bfqio_cgroup - bfq cgroup data structure.
  * @css: subsystem state for bfq in the containing cgroup.
@@ -517,6 +523,8 @@ extern void elv_fq_unset_request_ioq(str
 extern struct io_queue *elv_lookup_ioq_current(struct request_queue *q);
 extern struct io_queue *elv_lookup_ioq_bio(struct request_queue *q,
 						struct bio *bio);
+extern struct request_list *io_group_get_request_list(struct request_queue *q,
+						struct bio *bio);
 
 /* Returns single ioq associated with the io group. */
 static inline struct io_queue *io_group_ioq(struct io_group *iog)

Thanks
Vivek

Quoted 619 lines

---

block/blk-core.c    |   36 +++++++--
block/blk-sysfs.c   |   22 ++++--
block/elevator-fq.c |  133 ++++++++++++++++++++++++++++++++--
block/elevator-fq.h |  201 +++++++++++++++++++++++++++++++++++++++++++++++++++
4 files changed, 371 insertions(+), 21 deletions(-)
block/blk-core.c +28 -8
index 29bcfac..21023f7 100644
@@ -705,11 +705,15 @@ static void ioc_set_batching(struct request_queue *q, struct io_context *ioc)
 static void __freed_request(struct request_queue *q, int rw)
 {
 	struct request_list *rl = &q->rq;
-
-	if (rl->count[rw] < queue_congestion_off_threshold(q))
+	struct io_group *congested_iog, *full_iog;
+	
+	congested_iog = io_congested_io_group(q, rw);
+	if (rl->count[rw] < queue_congestion_off_threshold(q) &&
+	    !congested_iog)
 		blk_clear_queue_congested(q, rw);
 
-	if (rl->count[rw] + 1 <= q->nr_requests) {
+	full_iog = io_full_io_group(q, rw);
+	if (rl->count[rw] + 1 <= q->nr_requests && !full_iog) {
 		if (waitqueue_active(&rl->wait[rw]))
 			wake_up(&rl->wait[rw]);
 
@@ -721,13 +725,16 @@ static void __freed_request(struct request_queue *q, int rw)
 * A request has just been released.  Account for it, update the full and
 * congestion status, wake up any waiters.   Called under q->queue_lock.
 */
-static void freed_request(struct request_queue *q, int rw, int priv)
+static void freed_request(struct request_queue *q, struct io_group *iog,
+			  int rw, int priv)
 {
 	struct request_list *rl = &q->rq;
 
 	rl->count[rw]--;
 	if (priv)
 		rl->elvpriv--;
+	if (iog)
+		io_group_dec_count(iog, rw);
 
 	__freed_request(q, rw);
 
@@ -746,16 +753,21 @@ static struct request *get_request(struct request_queue *q, int rw_flags,
 {
 	struct request *rq = NULL;
 	struct request_list *rl = &q->rq;
+	struct io_group *iog;
 	struct io_context *ioc = NULL;
 	const int rw = rw_flags & 0x01;
 	int may_queue, priv;
 
+	iog = __io_get_io_group(q);
+
 	may_queue = elv_may_queue(q, rw_flags);
 	if (may_queue == ELV_MQUEUE_NO)
 		goto rq_starved;
 
-	if (rl->count[rw]+1 >= queue_congestion_on_threshold(q)) {
-		if (rl->count[rw]+1 >= q->nr_requests) {
+	if (rl->count[rw]+1 >= queue_congestion_on_threshold(q) ||
+	    io_group_congestion_on(iog, rw)) {
+		if (rl->count[rw]+1 >= q->nr_requests ||
+		    io_group_full(iog, rw)) {
 			ioc = current_io_context(GFP_ATOMIC, q->node);
 			/*
 			 * The queue will fill after this allocation, so set
@@ -789,8 +801,15 @@ static struct request *get_request(struct request_queue *q, int rw_flags,
 	if (rl->count[rw] >= (3 * q->nr_requests / 2))
 		goto out;
 
+	if (iog)
+		if (io_group_count(iog, rw) >=
+		   (3 * io_group_nr_requests(iog) / 2))
+			goto out;
+
 	rl->count[rw]++;
 	rl->starved[rw] = 0;
+	if (iog)
+		io_group_inc_count(iog, rw);
 
 	priv = !test_bit(QUEUE_FLAG_ELVSWITCH, &q->queue_flags);
 	if (priv)
@@ -808,7 +827,7 @@ static struct request *get_request(struct request_queue *q, int rw_flags,
 		 * wait queue, but this is pretty rare.
 		 */
 		spin_lock_irq(q->queue_lock);
-		freed_request(q, rw, priv);
+		freed_request(q, iog, rw, priv);
 
 		/*
 		 * in the very unlikely event that allocation failed and no
@@ -1073,12 +1092,13 @@ void __blk_put_request(struct request_queue *q, struct request *req)
 	if (req->cmd_flags & REQ_ALLOCED) {
 		int rw = rq_data_dir(req);
 		int priv = req->cmd_flags & REQ_ELVPRIV;
+		struct io_group *iog = io_request_io_group(req);
 
 		BUG_ON(!list_empty(&req->queuelist));
 		BUG_ON(!hlist_unhashed(&req->hash));
 
 		blk_free_request(q, req);
-		freed_request(q, rw, priv);
+		freed_request(q, iog, rw, priv);
 	}
 }
 EXPORT_SYMBOL_GPL(__blk_put_request);
block/blk-sysfs.c +14 -8
index 0d98c96..af5191c 100644
@@ -40,6 +40,7 @@ queue_requests_store(struct request_queue *q, const char *page, size_t count)
 {
 	struct request_list *rl = &q->rq;
 	unsigned long nr;
+	int iog_congested[2], iog_full[2];
 	int ret = queue_var_store(&nr, page, count);
 	if (nr < BLKDEV_MIN_RQ)
 		nr = BLKDEV_MIN_RQ;
@@ -47,27 +48,32 @@ queue_requests_store(struct request_queue *q, const char *page, size_t count)
 	spin_lock_irq(q->queue_lock);
 	q->nr_requests = nr;
 	blk_queue_congestion_threshold(q);
+	io_group_set_nrq_all(q, nr, iog_congested, iog_full);
 
-	if (rl->count[READ] >= queue_congestion_on_threshold(q))
+	if (rl->count[READ] >= queue_congestion_on_threshold(q) ||
+	    iog_congested[READ])
 		blk_set_queue_congested(q, READ);
-	else if (rl->count[READ] < queue_congestion_off_threshold(q))
+	else if (rl->count[READ] < queue_congestion_off_threshold(q) &&
+		 !iog_congested[READ])
 		blk_clear_queue_congested(q, READ);
 
-	if (rl->count[WRITE] >= queue_congestion_on_threshold(q))
+	if (rl->count[WRITE] >= queue_congestion_on_threshold(q) ||
+	    iog_congested[WRITE])
 		blk_set_queue_congested(q, WRITE);
-	else if (rl->count[WRITE] < queue_congestion_off_threshold(q))
+	else if (rl->count[WRITE] < queue_congestion_off_threshold(q) &&
+		 !iog_congested[WRITE])
 		blk_clear_queue_congested(q, WRITE);
 
-	if (rl->count[READ] >= q->nr_requests) {
+	if (rl->count[READ] >= q->nr_requests || iog_full[READ]) {
 		blk_set_queue_full(q, READ);
-	} else if (rl->count[READ]+1 <= q->nr_requests) {
+	} else if (rl->count[READ]+1 <= q->nr_requests && !iog_full[READ]) {
 		blk_clear_queue_full(q, READ);
 		wake_up(&rl->wait[READ]);
 	}
 
-	if (rl->count[WRITE] >= q->nr_requests) {
+	if (rl->count[WRITE] >= q->nr_requests || iog_full[WRITE]) {
 		blk_set_queue_full(q, WRITE);
-	} else if (rl->count[WRITE]+1 <= q->nr_requests) {
+	} else if (rl->count[WRITE]+1 <= q->nr_requests && !iog_full[WRITE]) {
 		blk_clear_queue_full(q, WRITE);
 		wake_up(&rl->wait[WRITE]);
 	}
block/elevator-fq.c +128 -5
index df53418..3b021f3 100644
@@ -924,6 +924,111 @@ struct io_group *io_lookup_io_group_current(struct request_queue *q)
 }
 EXPORT_SYMBOL(io_lookup_io_group_current);
 
+/*
+ * TODO
+ * This is complete dupulication of blk_queue_congestion_threshold()
+ * except for the argument type and name.  Can we merge them?
+ */
+static void io_group_nrq_congestion_threshold(struct io_group_nrq *nrq)
+{
+	int nr;
+
+	nr = nrq->nr_requests - (nrq->nr_requests / 8) + 1;
+	if (nr > nrq->nr_requests)
+		nr = nrq->nr_requests;
+	nrq->nr_congestion_on = nr;
+
+	nr = nrq->nr_requests - (nrq->nr_requests / 8)
+		- (nrq->nr_requests / 16) - 1;
+	if (nr < 1)
+		nr = 1;
+	nrq->nr_congestion_off = nr;
+}
+
+static void io_group_set_nrq(struct io_group_nrq *nrq, int nr_requests,
+			 int *congested, int *full)
+{
+	int i;
+
+	BUG_ON(nr_requests < 0);
+
+	nrq->nr_requests = nr_requests;
+	io_group_nrq_congestion_threshold(nrq);
+
+	for (i=0; i<2; i++) {
+		if (nrq->count[i] >= nrq->nr_congestion_on)
+			congested[i] = 1;
+		else if (nrq->count[i] < nrq->nr_congestion_off)
+			congested[i] = 0;
+
+		if (nrq->count[i] >= nrq->nr_requests)
+			full[i] = 1;
+		else if (nrq->count[i]+1 <= nrq->nr_requests)
+			full[i] = 0;
+	}
+}
+
+void io_group_set_nrq_all(struct request_queue *q, int nr,
+			    int *congested, int *full)
+{
+	struct elv_fq_data *efqd = &q->elevator->efqd;
+	struct hlist_head *head = &efqd->group_list;
+	struct io_group *root = efqd->root_group;
+	struct hlist_node *n;
+	struct io_group *iog;
+	struct io_group_nrq *nrq;
+	int nrq_congested[2];
+	int nrq_full[2];
+	int i;
+
+	for (i=0; i<2; i++)
+		*(congested + i) = *(full + i) = 0;
+
+	nrq = &root->nrq;
+	io_group_set_nrq(nrq, nr, nrq_congested, nrq_full);
+	for (i=0; i<2; i++) {
+		*(congested + i) |= nrq_congested[i];
+		*(full + i) |= nrq_full[i];
+	}
+
+	hlist_for_each_entry(iog, n, head, elv_data_node) {
+		nrq = &iog->nrq;
+		io_group_set_nrq(nrq, nr, nrq_congested, nrq_full);
+		for (i=0; i<2; i++) {
+			*(congested + i) |= nrq_congested[i];
+			*(full + i) |= nrq_full[i];
+		}
+	}
+}
+
+struct io_group *io_congested_io_group(struct request_queue *q, int rw)
+{
+	struct hlist_head *head = &q->elevator->efqd.group_list;
+	struct hlist_node *n;
+	struct io_group *iog;
+
+	hlist_for_each_entry(iog, n, head, elv_data_node) {
+		struct io_group_nrq *nrq = &iog->nrq;
+		if (nrq->count[rw] >= nrq->nr_congestion_off)
+			return iog;
+	}
+	return NULL;
+}
+
+struct io_group *io_full_io_group(struct request_queue *q, int rw)
+{
+	struct hlist_head *head = &q->elevator->efqd.group_list;
+	struct hlist_node *n;
+	struct io_group *iog;
+
+	hlist_for_each_entry(iog, n, head, elv_data_node) {
+		struct io_group_nrq *nrq = &iog->nrq;
+		if (nrq->count[rw] >= nrq->nr_requests)
+			return iog;
+	}
+	return NULL;
+}
+
 void io_group_init_entity(struct io_cgroup *iocg, struct io_group *iog)
 {
 	struct io_entity *entity = &iog->entity;
@@ -934,6 +1039,12 @@ void io_group_init_entity(struct io_cgroup *iocg, struct io_group *iog)
 	entity->my_sched_data = &iog->sched_data;
 }
 
+static void io_group_init_nrq(struct request_queue *q, struct io_group_nrq *nrq)
+{
+	nrq->nr_requests = q->nr_requests;
+	io_group_nrq_congestion_threshold(nrq);
+}
+
 void io_group_set_parent(struct io_group *iog, struct io_group *parent)
 {
 	struct io_entity *entity;
@@ -1053,6 +1164,8 @@ struct io_group *io_group_chain_alloc(struct request_queue *q, void *key,
 		io_group_init_entity(iocg, iog);
 		iog->my_entity = &iog->entity;
 
+		io_group_init_nrq(q, &iog->nrq);
+
 		if (leaf == NULL) {
 			leaf = iog;
 			prev = leaf;
@@ -1176,7 +1289,7 @@ struct io_group *io_find_alloc_group(struct request_queue *q,
 * Generic function to make sure cgroup hierarchy is all setup once a request
 * from a cgroup is received by the io scheduler.
 */
-struct io_group *io_get_io_group(struct request_queue *q)
+struct io_group *__io_get_io_group(struct request_queue *q)
 {
 	struct cgroup *cgroup;
 	struct io_group *iog;
@@ -1192,6 +1305,19 @@ struct io_group *io_get_io_group(struct request_queue *q)
 	return iog;
 }
 
+struct io_group *io_get_io_group(struct request_queue *q)
+{
+	struct io_group *iog;
+	unsigned long flags;
+
+	spin_lock_irqsave(q->queue_lock, flags);
+	iog = __io_get_io_group(q);
+	spin_unlock_irqrestore(q->queue_lock, flags);
+	BUG_ON(!iog);
+
+	return iog;
+}
+
 void io_free_root_group(struct elevator_queue *e)
 {
 	struct io_cgroup *iocg = &io_root_cgroup;
@@ -1220,6 +1346,7 @@ struct io_group *io_alloc_root_group(struct request_queue *q,
 	iog->entity.parent = NULL;
 	for (i = 0; i < IO_IOPRIO_CLASSES; i++)
 		iog->sched_data.service_tree[i] = IO_SERVICE_TREE_INIT;
+	io_group_init_nrq(q, &iog->nrq);
 
 	iocg = &io_root_cgroup;
 	spin_lock_irq(&iocg->lock);
@@ -1533,15 +1660,11 @@ void elv_fq_set_request_io_group(struct request_queue *q,
 						struct request *rq)
 {
 	struct io_group *iog;
-	unsigned long flags;
 
 	/* Make sure io group hierarchy has been setup and also set the
 	 * io group to which rq belongs. Later we should make use of
 	 * bio cgroup patches to determine the io group */
-	spin_lock_irqsave(q->queue_lock, flags);
 	iog = io_get_io_group(q);
-	spin_unlock_irqrestore(q->queue_lock, flags);
-	BUG_ON(!iog);
 
 	/* Store iog in rq. TODO: take care of referencing */
 	rq->iog = iog;
block/elevator-fq.h +201 -0
index fc4110d..f8eabd4 100644
@@ -187,6 +187,22 @@ struct io_queue {
 
 #ifdef CONFIG_GROUP_IOSCHED
 /**
+ * struct io_group_nrq - structure to store allocated requests info
+ * @nr_requests: maximun num of requests for the io_group
+ * @nr_congestion_on: threshold to determin the io_group is cogested.
+ * @nr_congestion_off: threshold to determin the io_group is not congested.
+ * @count: num of allocated requests.
+ *
+ * All fields are protected by queue_lock.
+ */
+struct io_group_nrq {
+	unsigned long nr_requests;
+	unsigned int nr_congestion_on;
+	unsigned int nr_congestion_off;
+	int count[2];
+};
+
+/**
 * struct bfq_group - per (device, cgroup) data structure.
 * @entity: schedulable entity to insert into the parent group sched_data.
 * @sched_data: own sched_data, to contain child entities (they may be
@@ -235,6 +251,8 @@ struct io_group {
 
 	/* Single ioq per group, used for noop, deadline, anticipatory */
 	struct io_queue *ioq;
+
+	struct io_group_nrq nrq;
 };
 
 /**
@@ -469,6 +487,11 @@ extern int elv_fq_set_request_ioq(struct request_queue *q, struct request *rq,
 extern void elv_fq_unset_request_ioq(struct request_queue *q,
 					struct request *rq);
 extern struct io_queue *elv_lookup_ioq_current(struct request_queue *q);
+extern void io_group_set_nrq_all(struct request_queue *q, int nr,
+			    int *congested, int *full);
+extern struct io_group *io_congested_io_group(struct request_queue *q, int rw);
+extern struct io_group *io_full_io_group(struct request_queue *q, int rw);
+extern struct io_group *__io_get_io_group(struct request_queue *q);
 
 /* Returns single ioq associated with the io group. */
 static inline struct io_queue *io_group_ioq(struct io_group *iog)
@@ -486,6 +509,52 @@ static inline void io_group_set_ioq(struct io_group *iog, struct io_queue *ioq)
 	iog->ioq = ioq;
 }
 
+static inline struct io_group *io_request_io_group(struct request *rq)
+{
+	return rq->iog;
+}
+
+static inline unsigned long io_group_nr_requests(struct io_group *iog)
+{
+	BUG_ON(!iog);
+	return iog->nrq.nr_requests;
+}
+
+static inline int io_group_inc_count(struct io_group *iog, int rw)
+{
+	BUG_ON(!iog);
+	return iog->nrq.count[rw]++;
+}
+
+static inline int io_group_dec_count(struct io_group *iog, int rw)
+{
+	BUG_ON(!iog);
+	return iog->nrq.count[rw]--;
+}
+
+static inline int io_group_count(struct io_group *iog, int rw)
+{
+	BUG_ON(!iog);
+	return iog->nrq.count[rw];
+}
+
+static inline int io_group_congestion_on(struct io_group *iog, int rw)
+{
+	BUG_ON(!iog);
+	return iog->nrq.count[rw] + 1 >= iog->nrq.nr_congestion_on;
+}
+
+static inline int io_group_congestion_off(struct io_group *iog, int rw)
+{
+	BUG_ON(!iog);
+	return iog->nrq.count[rw] < iog->nrq.nr_congestion_off;
+}
+
+static inline int io_group_full(struct io_group *iog, int rw)
+{
+	BUG_ON(!iog);
+	return iog->nrq.count[rw] + 1 >= iog->nrq.nr_requests;
+}
 #else /* !GROUP_IOSCHED */
 /*
 * No ioq movement is needed in case of flat setup. root io group gets cleaned
@@ -537,6 +606,71 @@ static inline struct io_queue *elv_lookup_ioq_current(struct request_queue *q)
 	return NULL;
 }
 
+static inline void io_group_set_nrq_all(struct request_queue *q, int nr,
+					int *congested, int *full)
+{
+	int i;
+	for (i=0; i<2; i++)
+		*(congested + i) = *(full + i) = 0;
+}
+
+static inline struct io_group *
+io_congested_io_group(struct request_queue *q, int rw)
+{
+	return NULL;
+}
+
+static inline struct io_group *
+io_full_io_group(struct request_queue *q, int rw)
+{
+	return NULL;
+}
+
+static inline struct io_group *__io_get_io_group(struct request_queue *q)
+{
+	return NULL;
+}
+
+static inline struct io_group *io_request_io_group(struct request *rq)
+{
+	return NULL;
+}
+
+static inline unsigned long io_group_nr_requests(struct io_group *iog)
+{
+	return 0;
+}
+
+static inline int io_group_inc_count(struct io_group *iog, int rw)
+{
+	return 0;
+}
+
+static inline int io_group_dec_count(struct io_group *iog, int rw)
+{
+	return 0;
+}
+
+static inline int io_group_count(struct io_group *iog, int rw)
+{
+	return 0;
+}
+
+static inline int io_group_congestion_on(struct io_group *iog, int rw)
+{
+	return 0;
+}
+
+static inline int io_group_congestion_off(struct io_group *iog, int rw)
+{
+	return 1;
+}
+
+static inline int io_group_full(struct io_group *iog, int rw)
+{
+	return 0;
+}
+
 #endif /* GROUP_IOSCHED */
 
 /* Functions used by blksysfs.c */
@@ -589,6 +723,9 @@ extern void elv_free_ioq(struct io_queue *ioq);
 
 #else /* CONFIG_ELV_FAIR_QUEUING */
 
+struct io_group {
+};
+
 static inline int elv_init_fq_data(struct request_queue *q,
 					struct elevator_queue *e)
 {
@@ -655,5 +792,69 @@ static inline struct io_queue *elv_lookup_ioq_current(struct request_queue *q)
 	return NULL;
 }
 
+static inline void io_group_set_nrq_all(struct request_queue *q, int nr,
+					int *congested, int *full)
+{
+	int i;
+	for (i=0; i<2; i++)
+		*(congested + i) = *(full + i) = 0;
+}
+
+static inline struct io_group *
+io_congested_io_group(struct request_queue *q, int rw)
+{
+	return NULL;
+}
+
+static inline struct io_group *
+io_full_io_group(struct request_queue *q, int rw)
+{
+	return NULL;
+}
+
+static inline struct io_group *__io_get_io_group(struct request_queue *q)
+{
+	return NULL;
+}
+
+static inline struct io_group *io_request_io_group(struct request *rq)
+{
+	return NULL;
+}
+
+static inline unsigned long io_group_nr_requests(struct io_group *iog)
+{
+	return 0;
+}
+
+static inline int io_group_inc_count(struct io_group *iog, int rw)
+{
+	return 0;
+}
+
+static inline int io_group_dec_count(struct io_group *iog, int rw)
+{
+	return 0;
+}
+
+static inline int io_group_count(struct io_group *iog, int rw)
+{
+	return 0;
+}
+
+static inline int io_group_congestion_on(struct io_group *iog, int rw)
+{
+	return 0;
+}
+
+static inline int io_group_congestion_off(struct io_group *iog, int rw)
+{
+	return 1;
+}
+
+static inline int io_group_full(struct io_group *iog, int rw)
+{
+	return 0;
+}
 #endif /* CONFIG_ELV_FAIR_QUEUING */
 #endif /* _BFQ_SCHED_H */
Signature · 7 lines
1.5.4.3


-- 
IKEDA, Munehiro
NEC Corporation of America
  m-ikeda-MDRzhb/z0dd8UrSeD/[email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]