IO scheduler based io controller (V5)
Subsystems (19)
Hi All,
Here is the V5 of the IO controller patches generated on top of 2.6.30.
Previous versions of the patches was posted here.
(V1) http://lkml.org/lkml/2009/3/11/486
(V2) http://lkml.org/lkml/2009/5/5/275
(V3) http://lkml.org/lkml/2009/5/26/472
(V4) http://lkml.org/lkml/2009/6/8/580
This patchset is still work in progress but I want to keep on getting the snapshot of my tree out at regular intervals to get the feedback hence V5.
Changes from V4
===============
- Implemented bdi_*_congested_group() functions to also determine if a particular io group on a bdi is congested or not. So far we only used determine whether bdi is congested or not. But now there is one request list per group and one also needs to check whether the particular io group io is going into is congested or not.
- Fixed preemption logic in hiearchical mode. In hiearchical mode, one needs to traverse up the hiearchy so that current queue and new queue are at same level to make a decision whether preeption should be done or not. Took the idea and code from CFS cpu scheduler.
- There were some tunables which were appearing under /sys/block/<device>/queue dir but these tunables actually belonged to ioschedulers in hierarhical moded. Fixed it.
- Fixed another preemption issue where if any RT queue was pending (busy_rt_queues), current queue was being expired. Now this preemption is done only if there are busy_rt_queues in the same group.
(Though I think that busy_rt_queues is redundant code as the moment RT
request comes, we preempt the BE queue so we should never run into the issue of RT reuqest pending while BE is running. Keeping the code for the time being).
- Applied the patch from Gui where he got rid of only_root_group code and now used cgroups children list to determine if root group is only group or there are childrens too.
- Applied few cleanup patches from Gui.
- We store the device id (major, minor) in io group. Previously I was retrieving that info from bio. Switched to gettting that info from backing device.
Limitations
===========
- This IO controller provides the bandwidth control at the IO scheduler level (leaf node in stacked hiearchy of logical devices). So there can be cases (depending on configuration) where application does not see proportional BW division at higher logical level device.
LWN has written an article about the issue here.
http://lwn.net/Articles/332839/
How to solve the issue of fairness at higher level logical devices
==================================================================
Couple of suggestions have come forward.
- Implement IO control at IO scheduler layer and then with the help of some daemon, adjust the weight on underlying devices dynamiclly, depending on what kind of BW gurantees are to be achieved at higher level logical block devices.
- Also implement a higher level IO controller along with IO scheduler based controller and let user choose one depending on his needs.
A higher level controller does not know about the assumptions/policies of unerldying IO scheduler, hence it has the potential to break down the IO scheduler's policy with-in cgroup. A lower level controller can work with IO scheduler much more closely and efficiently.
Other active IO controller developments
=======================================
IO throttling
-------------
This is a max bandwidth controller and not the proportional one. Secondly it is a second level controller which can break the IO scheduler's policy/assumtions with-in cgroup.
dm-ioband
---------
This is a proportional bandwidth controller implemented as device mapper driver. It is also a second level controller which can break the IO scheduler's policy/assumptions with-in cgroup.
Testing
=======
I have been able to do only very basic testing of reads and writes.
Test1 (Fairness for synchronous reads)
======================================
- Two dd in two cgroups with cgrop weights 1000 and 500. Ran two "dd" in those cgroups (With CFQ scheduler and /sys/block/<device>/queue/fairness = 1)
dd if=/mnt/$BLOCKDEV/zerofile1 of=/dev/null &
dd if=/mnt/$BLOCKDEV/zerofile2 of=/dev/null &
234179072 bytes (234 MB) copied, 3.9065 s, 59.9 MB/s 234179072 bytes (234 MB) copied, 5.19232 s, 45.1 MB/s
group1 time=8 16 2471 group1 sectors=8 16 457840
group2 time=8 16 1220 group2 sectors=8 16 225736
First two fields in time and sectors statistics represent major and minor number of the device. Third field represents disk time in milliseconds and number of sectors transferred respectively.
This patchset tries to provide fairness in terms of disk time received. group1 got almost double of group2 disk time (At the time of first dd finish). These time and sectors statistics can be read using io.disk_time and io.disk_sector files in cgroup. More about it in documentation file.
Test2 (Fairness for async writes)
=================================
Fairness for async writes is tricky and biggest reason is that async writes are cached in higher layers (page cahe) as well as possibly in file system layer also (btrfs, xfs etc), and are dispatched to lower layers not necessarily in proportional manner.
For example, consider two dd threads reading /dev/zero as input file and doing writes of huge files. Very soon we will cross vm_dirty_ratio and dd thread will be forced to write out some pages to disk before more pages can be dirtied. But not necessarily dirty pages of same thread are picked. It can very well pick the inode of lesser priority dd thread and do some writeout. So effectively higher weight dd is doing writeouts of lower weight dd pages and we don't see service differentation.
IOW, the core problem with async write fairness is that higher weight thread does not throw enought IO traffic at IO controller to keep the queue continuously backlogged. In my testing, there are many .2 to .8 second intervals where higher weight queue is empty and in that duration lower weight queue get lots of job done giving the impression that there was no service differentiation.
In summary, from IO controller point of view async writes support is there. Because page cache has not been designed in such a manner that higher prio/weight writer can do more write out as compared to lower prio/weight writer, gettting service differentiation is hard and it is visible in some cases and not visible in some cases.
To get fairness for async writes in all cases, higher layer needs to be fixed. That probably is a lot of work. Do we really care that much for fairness among two writer cgroups? One can choose to do direct IO if fairness for buffered writes really matters for him. I think we care more for fairness in following cases and with this patch we should be able to achive that.
- Read Vs Read
- Read Vs Writes (Buffered writes or direct IO writes)
- Making sure that isolation is achieved between reader and writer cgroup.
- All form of direct IO.
Following is the only case where it is hard to ensure fairness between cgroups because of higher layer design.
- Buffered writes Vs Buffered Writes.
So to test async writes I generated lots of write traffic in two cgroups (50 fio threads) and watched the disk time statistics in respective cgroups at the interval of 2 seconds. Thanks to ryo tsuruta for the test case.
*****************************************************************
sync
echo 3 > /proc/sys/vm/drop_caches
fio_args="--size=64m --rw=write --numjobs=50 --group_reporting"
echo $$ > /cgroup/bfqio/test1/tasks
fio $fio_args --name=test1 --directory=/mnt/sdd1/fio/ --output=/mnt/sdd1/fio/test1.log &
echo $$ > /cgroup/bfqio/test2/tasks
fio $fio_args --name=test2 --directory=/mnt/sdd2/fio/ --output=/mnt/sdd2/fio/test2.log &
***********************************************************************
And watched the disk time and sector statistics for the both the cgroups every 2 seconds using a script. How is snippet from output.
test1 statistics: time=8 48 1315 sectors=8 48 55776 dq=8 48 1
test2 statistics: time=8 48 633 sectors=8 48 14720 dq=8 48 2
test1 statistics: time=8 48 5586 sectors=8 48 339064 dq=8 48 2
test2 statistics: time=8 48 2985 sectors=8 48 146656 dq=8 48 3
test1 statistics: time=8 48 9935 sectors=8 48 628728 dq=8 48 3
test2 statistics: time=8 48 5265 sectors=8 48 278688 dq=8 48 4
test1 statistics: time=8 48 14156 sectors=8 48 932488 dq=8 48 6
test2 statistics: time=8 48 7646 sectors=8 48 412704 dq=8 48 7
test1 statistics: time=8 48 18141 sectors=8 48 1231488 dq=8 48 10
test2 statistics: time=8 48 9820 sectors=8 48 548400 dq=8 48 8
test1 statistics: time=8 48 21953 sectors=8 48 1485632 dq=8 48 13
test2 statistics: time=8 48 12394 sectors=8 48 698288 dq=8 48 10
test1 statistics: time=8 48 25167 sectors=8 48 1705264 dq=8 48 13
test2 statistics: time=8 48 14042 sectors=8 48 817808 dq=8 48 10
First two fields in time and sectors statistics represent major and minor number of the device. Third field represents disk time in milliseconds and number of sectors transferred respectively.
So disk time consumed by group1 is almost double of group2.
TODO
====
- Lots of code cleanups, testing, bug fixing, optimizations, benchmarking etc...
- Work on a better interface (possibly cgroup based) for configuring per group request descriptor limits.
- Debug and fix some of the areas like page cache where higher weight cgroup async writes are stuck behind lower weight cgroup async writes.
Thanks
Vivek
PATCH 1/20 io-controller: Documentation
o Documentation for io-controller.
--- Documentation/block/00-INDEX | 2 + Documentation/block/io-controller.txt | 360 +++++++++++++++++++++++++++++++++ 2 files changed, 362 insertions(+), 0 deletions(-) create mode 100644 Documentation/block/io-controller.txt
Documentation/block/00-INDEX +2 -0
@@ -10,6 +10,8 @@ capability.txt - Generic Block Device Capability (/sys/block/<disk>/capability) deadline-iosched.txt - Deadline IO scheduler tunables +io-controller.txt + - IO controller for provding hierarchical IO scheduling ioprio.txt - Block io priorities (in CFQ scheduler) request.txt
Documentation/block/io-controller.txt new file +360 -0
360 lines not on this page · raw message
Signature · 1 line
1.6.0.6
A message that is not in the archive (sent off-list or not yet received): 1245443858-8487-1-git-send-email-vgoyal-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org
PATCH 1/20 io-controller: Documentation
o Documentation for io-controller.
--- Documentation/block/00-INDEX | 2 + Documentation/block/io-controller.txt | 360 +++++++++++++++++++++++++++++++++ 2 files changed, 362 insertions(+), 0 deletions(-) create mode 100644 Documentation/block/io-controller.txt
Documentation/block/00-INDEX +2 -0
@@ -10,6 +10,8 @@ capability.txt - Generic Block Device Capability (/sys/block/<disk>/capability) deadline-iosched.txt - Deadline IO scheduler tunables +io-controller.txt + - IO controller for provding hierarchical IO scheduling ioprio.txt - Block io priorities (in CFQ scheduler) request.txt
Documentation/block/io-controller.txt new file +360 -0
360 lines not on this page · raw message
Signature · 1 line
1.6.0.6
PATCH 2/20 io-controller: Common flat fair queuing code in elevaotor layer
In reply to a message not in the archiveThis is common fair queuing code in elevator layer. This is controlled by config option CONFIG_ELV_FAIR_QUEUING. This patch initially only introduces flat fair queuing support where there is only one group, "root group" and all the tasks belong to root group.
This elevator layer changes are backward compatible. That means any ioscheduler using old interfaces will continue to work.
This code is essentially the CFQ code for fair queuing. The primary difference is that flat rounding robin algorithm of CFQ has been replaced with BFQ (WF2Q+).
--- block/Kconfig.iosched | 13 + block/Makefile | 1 + block/elevator-fq.c | 2015 ++++++++++++++++++++++++++++++++++++++++++++++ block/elevator-fq.h | 473 +++++++++++ block/elevator.c | 46 +- include/linux/blkdev.h | 5 + include/linux/elevator.h | 51 ++ 7 files changed, 2593 insertions(+), 11 deletions(-) create mode 100644 block/elevator-fq.c create mode 100644 block/elevator-fq.h
block/Kconfig.iosched +13 -0
@@ -2,6 +2,19 @@ if BLOCK menu "IO Schedulers" +config ELV_FAIR_QUEUING + bool "Elevator Fair Queuing Support" + default n + ---help--- + Traditionally only cfq had notion of multiple queues and it did + fair queuing at its own. With the cgroups and need of controlling + IO, now even the simple io schedulers like noop, deadline, as will + have one queue per cgroup and will need hierarchical fair queuing. + Instead of every io scheduler implementing its own fair queuing + logic, this option enables fair queuing in elevator layer so that + other ioschedulers can make use of it. + If unsure, say N. + config IOSCHED_NOOP bool default y
block/Makefile +1 -0
@@ -15,3 +15,4 @@ obj-$(CONFIG_IOSCHED_CFQ) += cfq-iosched.o obj-$(CONFIG_BLOCK_COMPAT) += compat_ioctl.o obj-$(CONFIG_BLK_DEV_INTEGRITY) += blk-integrity.o +obj-$(CONFIG_ELV_FAIR_QUEUING) += elevator-fq.o
block/elevator-fq.c new file +2015 -0
2,015 lines not on this page · raw message
block/elevator-fq.h new file +473 -0
473 lines not on this page · raw message
block/elevator.c +35 -11
116 lines not on this page · raw message
include/linux/blkdev.h +5 -0
@@ -245,6 +245,11 @@ struct request {/* for bidi */ struct request *next_rq; + +#ifdef CONFIG_ELV_FAIR_QUEUING +/* io queue request belongs to */ +struct io_queue *ioq; +#endif };static inline unsigned short req_get_ioprio (struct request *req)
include/linux/elevator.h +51 -0
86 lines not on this page · raw message
Signature · 1 line
1.6.0.6
PATCH 3/20 io-controller: Charge for time slice based on average disk rate
In reply to a message not in the archiveo There are situations where a queue gets expired very soon and it looks as if time slice used by that queue is zero. For example, If an async queue dispatches a bunch of requests and queue is expired before first request completes. Another example is where a queue is expired as soon as first request completes and queue has no more requests (sync queues on SSD).
o Currently we just charge 25% of slice length in such cases. This patch tries to improve on that approximation by keeping a track of average disk rate and charging for time by nr_sectors/disk_rate.
o This is still experimental, not very sure if it gives measurable improvement or not.
--- block/elevator-fq.c | 85 +++++++++++++++++++++++++++++++++++++++++++++++++- block/elevator-fq.h | 11 ++++++ 2 files changed, 94 insertions(+), 2 deletions(-)
block/elevator-fq.c +83 -2
@@ -21,6 +21,9 @@ const int elv_slice_async_rq = 2;int elv_slice_idle =HZ /125 ;static struct kmem_cache *elv_ioq_pool; +/* Maximum Window length for updating average disk rate */ +static int elv_rate_sampling_window =HZ /10 ; +#define ELV_SLICE_SCALE (5)#define ELV_HW_QUEUE_MIN (5)#define IO_SERVICE_TREE_INIT ((struct io_service_tree) \
@@ -961,6 +964,47 @@ static void elv_ioq_update_io_thinktime(struct io_queue *ioq) ioq->ttime_mean = (ioq->ttime_total +128 ) / ioq->ttime_samples; } +static void elv_update_io_rate (struct elv_fq_data *efqd,struct request *rq) +{ +long elapsed = jiffies - efqd->rate_sampling_start; +unsigned long total; + +/* sampling window is off */ +if (!efqd->rate_sampling_start) +return ; + + efqd->rate_sectors_current += rq->nr_sectors; + +if (efqd->rq_in_driver && (elapsed < elv_rate_sampling_window)) +return ; + + efqd->rate_sectors = (7 *efqd->rate_sectors + +256 *efqd->rate_sectors_current) /8 ; + +if (!elapsed) { +/* +* updating rate before a jiffy could complete. Could be a +* problem with fast queuing/non-queuing hardware. Should we +* look at higher resolution time source? +* +* In case of non-queuing hardware we will probably not try to +* dispatch from multiple queues and will be able to account +* for disk time used and will not need this approximation +* anyway? +*/ + elapsed =1 ; + } + + efqd->rate_time = (7 *efqd->rate_time +256 *elapsed) /8 ; + total = efqd->rate_sectors + (efqd->rate_time/2 ); + efqd->mean_rate = total/efqd->rate_time; + +elv_log (efqd,"mean_rate=%d, t=%d s=%d" , efqd->mean_rate, + elapsed, efqd->rate_sectors_current); + efqd->rate_sampling_start =0 ; + efqd->rate_sectors_current =0 ; +} +/* * Disable idle window if the process thinks too long. * This idle flag can also be updated by io scheduler.
@@ -1252,6 +1296,34 @@ void elv_del_ioq_busy(struct elevator_queue *e, struct io_queue *ioq, }/* +* Calculate the effective disk time used by the queue based on how many +* sectors queue has dispatched and what is the average disk rate +* Returns disk time in ms. +*/ +static inline unsigned long elv_disk_time_used (struct request_queue *q, +struct io_queue *ioq) +{ +struct elv_fq_data *efqd = &q->elevator->efqd; +struct io_entity *entity = &ioq->entity; +unsigned long jiffies_used =0 ; + +if (!efqd->mean_rate) +return entity->budget/4 ; + +/* Charge the queue based on average disk rate */ + jiffies_used = ioq->nr_sectors/efqd->mean_rate; + +if (!jiffies_used) + jiffies_used =1 ; + +elv_log_ioq (efqd, ioq,"disk time=%ldms sect=%lu rate=%ld" , +jiffies_to_msecs (jiffies_used), + ioq->nr_sectors, efqd->mean_rate); + +return jiffies_used; +} + +/* * Do the accounting. Determine how much service (in terms of time slices) * current queue used and adjust the start, finish time of queue and vtime * of the tree accordingly.
@@ -1303,7 +1375,7 @@ void __elv_ioq_slice_expired(struct request_queue *q, struct io_queue *ioq)* the requests to finish. But this will reduce throughput. */ if (!ioq->slice_end) - slice_used = entity->budget/4 ; + slice_used =elv_disk_time_used (q, ioq);else {if (time_after (ioq->slice_end, jiffies)) { slice_unused = ioq->slice_end - jiffies;
@@ -1313,7 +1385,7 @@ void __elv_ioq_slice_expired(struct request_queue *q, struct io_queue *ioq)* completing first request. Charge 25% of * slice. */ - slice_used = entity->budget/4 ; + slice_used =elv_disk_time_used (q, ioq); }else slice_used = entity->budget - slice_unused; }else {
@@ -1331,6 +1403,8 @@ void __elv_ioq_slice_expired(struct request_queue *q, struct io_queue *ioq)BUG_ON (ioq != efqd->active_queue);elv_reset_active_ioq (efqd); +/* Queue is being expired. Reset number of secotrs dispatched */ + ioq->nr_sectors =0 ;if (!ioq->nr_queued)elv_del_ioq_busy (q->elevator, ioq,1 );else
@@ -1664,6 +1738,7 @@ void elv_fq_dispatched_request(struct elevator_queue *e, struct request *rq)BUG_ON (!ioq);elv_ioq_request_dispatched (ioq); + ioq->nr_sectors += rq->nr_sectors;elv_ioq_request_removed (e, rq);elv_clear_ioq_must_dispatch (ioq); }
@@ -1676,6 +1751,10 @@ void elv_fq_activate_rq(struct request_queue *q, struct request *rq)return ; efqd->rq_in_driver++; + +if (!efqd->rate_sampling_start) + efqd->rate_sampling_start = jiffies; +elv_log_ioq (efqd,rq_ioq (rq),"activate rq, drv=%d" , efqd->rq_in_driver); }
@@ -1767,6 +1846,8 @@ void elv_ioq_completed_request(struct request_queue *q, struct request *rq) efqd->rq_in_driver--; ioq->dispatched--; +elv_update_io_rate (efqd, rq); +if (sync) ioq->last_end_request = jiffies;
block/elevator-fq.h +11 -0
@@ -169,6 +169,9 @@ struct io_queue {/* Requests dispatched from this queue */ int dispatched; +/* Number of sectors dispatched in current dispatch round */ +unsigned long nr_sectors; +/* Keep a track of think time of processes in this queue */ unsigned long last_end_request;unsigned long ttime_total;
@@ -225,6 +228,14 @@ struct elv_fq_data {struct work_struct unplug_work;unsigned int elv_slice[2 ]; + +/* Fields for keeping track of average disk rate */ +unsigned long rate_sectors;/* number of sectors finished */ +unsigned long rate_time;/* jiffies elapsed */ +unsigned long mean_rate;/* sectors per jiffy */ +unsigned long long rate_sampling_start;/*sampling window start jifies*/ +/* number of sectors finished io during current sampling window */ +unsigned long rate_sectors_current; };extern int elv_slice_idle;
Signature · 1 line
1.6.0.6
PATCH 4/20 io-controller: Modify cfq to make use of flat elevator fair queuing
In reply to a message not in the archiveThis patch changes cfq to use fair queuing code from elevator layer.
--- block/Kconfig.iosched | 3 +- block/cfq-iosched.c | 1106 +++++++++------------------------------------ include/linux/iocontext.h | 5 - 3 files changed, 222 insertions(+), 892 deletions(-)
block/Kconfig.iosched +2 -1
@@ -3,7 +3,7 @@ if BLOCK menu "IO Schedulers" config ELV_FAIR_QUEUING - bool "Elevator Fair Queuing Support" + bool default n ---help--- Traditionally only cfq had notion of multiple queues and it did
@@ -46,6 +46,7 @@ config IOSCHED_DEADLINE config IOSCHED_CFQ tristate "CFQ I/O scheduler" + select ELV_FAIR_QUEUING default y ---help--- The CFQ I/O scheduler tries to distribute bandwidth equally
block/cfq-iosched.c +220 -886
1,748 lines not on this page · raw message
include/linux/iocontext.h +0 -5
@@ -39,13 +39,8 @@ struct cfq_io_context {struct io_context *ioc; -unsigned long last_end_request;sector_t last_request_pos; -unsigned long ttime_total; -unsigned long ttime_samples; -unsigned long ttime_mean; -unsigned int seek_samples;u64 seek_total;sector_t seek_mean;
Signature · 1 line
1.6.0.6
PATCH 5/20 io-controller: Common hierarchical fair queuing code in elevaotor layer
In reply to a message not in the archiveo This patch enables hierarchical fair queuing in common layer. It is controlled by config option CONFIG_GROUP_IOSCHED.
o Requests keep a reference on ioq and ioq keeps keep a reference on groups. For async queues in CFQ, and single ioq in other schedulers, io_group also keeps are reference on io_queue. This reference on ioq is dropped when the queue is released (elv_release_ioq). So the queue can be freed.
When a queue is released, it puts the reference to io_group and the io_group is released after all the queues are released. Child groups also take reference on parent groups, and release it when they are destroyed.
o Reads of iocg->group_data are not always iocg->lock; so all the operations on that list are still protected by RCU. All modifications to iocg->group_data should always done under iocg->lock.
Whenever iocg->lock and queue_lock can both be held, queue_lock should be held first. This avoids all deadlocks. In order to avoid race between cgroup deletion and elevator switch the following algorithm is used:
- Cgroup deletion path holds iocg->lock and removes iog entry to iocg->group_data list. Then it drops iocg->lock, holds queue_lock and destroys iog. So in this path, we never hold iocg->lock and queue_lock at the same time. Also, since we remove iog from iocg->group_data under iocg->lock, we can't race with elevator switch.
- Elevator switch path does not remove iog from iocg->group_data list directly. It first hold iocg->lock, scans iocg->group_data again to see if iog is still there; it removes iog only if it finds iog there. Otherwise, cgroup deletion must have removed it from the list, and cgroup deletion is responsible for removing iog.
So the path which removes iog from iocg->group_data list does the final removal of iog by calling __io_destroy_group() function.
--- block/blk-ioc.c | 3 + block/cfq-iosched.c | 2 + block/elevator-fq.c | 1221 +++++++++++++++++++++++++++++++++++++---- block/elevator-fq.h | 169 ++++++- block/elevator.c | 4 + include/linux/blkdev.h | 2 +- include/linux/cgroup_subsys.h | 7 + include/linux/iocontext.h | 5 + init/Kconfig | 8 + 9 files changed, 1313 insertions(+), 108 deletions(-)
block/blk-ioc.c +3 -0
@@ -95,6 +95,9 @@ struct io_context *alloc_io_context(gfp_t gfp_flags, int node)spin_lock_init (&ret->lock); ret->ioprio_changed =0 ; ret->ioprio =0 ; +#ifdef CONFIG_GROUP_IOSCHED + ret->cgroup_changed =0 ; +#endif ret->last_waited = jiffies;/* doesn't matter... */ ret->nr_batch_requests =0 ;/* because this is 0 */ ret->aic =NULL ;
block/cfq-iosched.c +2 -0
@@ -1306,6 +1306,8 @@ alloc_ioq:elv_mark_ioq_sync (cfqq->ioq); } cfqq->pid = current->pid; +/* ioq reference on iog */ +elv_get_iog (iog);cfq_log_cfqq (cfqd, cfqq,"alloced" ); }
block/elevator-fq.c +1120 -101
1,423 lines not on this page · raw message
block/elevator-fq.h +163 -6
259 lines not on this page · raw message
block/elevator.c +4 -0
@@ -105,6 +105,10 @@ int elv_rq_merge_ok(struct request *rq, struct bio *bio)if (bio_integrity (bio) !=blk_integrity_rq (rq))return 0 ; +/* If rq and bio belongs to different groups, dont allow merging */ +if (!io_group_allow_merge (rq, bio)) +return 0 ; +if (!elv_iosched_allow_merge (rq, bio))return 0 ;
include/linux/blkdev.h +1 -1
@@ -249,7 +249,7 @@ struct request {#ifdef CONFIG_ELV_FAIR_QUEUING /* io queue request belongs to */ struct io_queue *ioq; -#endif +#endif /* ELV_FAIR_QUEUING */ };static inline unsigned short req_get_ioprio (struct request *req)
include/linux/cgroup_subsys.h +7 -0
@@ -60,3 +60,10 @@ SUBSYS(net_cls)#endif /* */ + +#ifdef CONFIG_GROUP_IOSCHED +SUBSYS(io) +#endif + +/* */ +
include/linux/iocontext.h +5 -0
@@ -68,6 +68,11 @@ struct io_context {unsigned short ioprio;unsigned short ioprio_changed; +#ifdef CONFIG_GROUP_IOSCHED +/* If task changes the cgroup, elevator processes it asynchronously */ +unsigned short cgroup_changed; +#endif +/* * For request batching */
init/Kconfig +8 -0
@@ -606,6 +606,14 @@ config CGROUP_MEM_RES_CTLR_SWAP Now, memory usage of swap_cgroup is 2 bytes per entry. If swap page size is 4096bytes, 512k per 1Gbytes of swap. +config GROUP_IOSCHED + bool "Group IO Scheduler" + depends on CGROUPS && ELV_FAIR_QUEUING + default n + ---help--- + This feature lets IO scheduler recognize task groups and control + disk bandwidth allocation to such task groups. + endif # CGROUPS config MM_OWNER
Signature · 1 line
1.6.0.6
PATCH 6/20 io-controller: cfq changes to use hierarchical fair queuing code in elevaotor layer
In reply to a message not in the archiveMake cfq hierarhical.
--- block/Kconfig.iosched | 8 ++++++ block/cfq-iosched.c | 68 ++++++++++++++++++++++++++++++++++++++++++++++-- init/Kconfig | 2 +- 3 files changed, 74 insertions(+), 4 deletions(-)
block/Kconfig.iosched +8 -0
@@ -54,6 +54,14 @@ config IOSCHED_CFQ working environment, suitable for desktop systems. This is the default I/O scheduler. +config IOSCHED_CFQ_HIER + bool "CFQ Hierarchical Scheduling support" + depends on IOSCHED_CFQ && CGROUPS + select GROUP_IOSCHED + default n + ---help--- + Enable hierarhical scheduling in cfq. + choice prompt "Default I/O scheduler" default DEFAULT_CFQ
block/cfq-iosched.c +65 -3
99 lines not on this page · raw message
init/Kconfig +1 -1
@@ -607,7 +607,7 @@ config CGROUP_MEM_RES_CTLR_SWAP size is 4096bytes, 512k per 1Gbytes of swap. config GROUP_IOSCHED - bool "Group IO Scheduler" + bool depends on CGROUPS && ELV_FAIR_QUEUING default n ---help---
Signature · 1 line
1.6.0.6
PATCH 7/20 io-controller: Export disk time used and nr sectors dipatched through cgroups
In reply to a message not in the archiveo This patch exports some statistics through cgroup interface. Two of the statistics currently exported are actual disk time assigned to the cgroup and actual number of sectors dispatched to disk on behalf of this cgroup.
--- block/elevator-fq.c | 87 +++++++++++++++++++++++++++++++++++++++++++++++--- block/elevator-fq.h | 10 ++++++ 2 files changed, 91 insertions(+), 6 deletions(-)
block/elevator-fq.c +81 -6
@@ -13,6 +13,7 @@#include <linux/blkdev.h> #include "elevator-fq.h" #include <linux/blktrace_api.h> +#include <linux/seq_file.h> /* Values taken from cfq */ const int elv_slice_sync =HZ /10 ;
@@ -979,12 +980,15 @@ struct io_entity *bfq_lookup_next_entity(struct io_sched_data *sd,return entity; } -void entity_served (struct io_entity *entity,bfq_service_t served) +void entity_served (struct io_entity *entity,bfq_service_t served, +bfq_service_t nr_sectors) {struct io_service_tree *st;for_each_entity (entity) { st =io_entity_service_tree (entity); entity->service += served; + entity->total_service += served; + entity->total_sector_service += nr_sectors;BUG_ON (st->wsum ==0 ); st->vtime +=bfq_delta (served, st->wsum);bfq_forget_idle (st);
@@ -1145,6 +1149,66 @@ STORE_FUNCTION(weight, 1, WEIGHT_MAX);STORE_FUNCTION (ioprio_class,IOPRIO_CLASS_RT ,IOPRIO_CLASS_IDLE );#undef STORE_FUNCTION +static int io_cgroup_disk_time_read (struct cgroup *cgroup, +struct cftype *cftype,struct seq_file *m) +{ +struct io_cgroup *iocg; +struct io_group *iog; +struct hlist_node *n; + +if (!cgroup_lock_live_group (cgroup)) +return -ENODEV ; + + iocg =cgroup_to_io_cgroup (cgroup); + +spin_lock_irq (&iocg->lock); +hlist_for_each_entry_rcu (iog, n, &iocg->group_data, group_node) { +/* +* There might be groups which are not functional and +* waiting to be reclaimed upon cgoup deletion. +*/ +if (iog->key) { +seq_printf (m,"%u %u %lu\n" ,MAJOR (iog->dev), +MINOR (iog->dev), + iog->entity.total_service); + } + } +spin_unlock_irq (&iocg->lock); +cgroup_unlock (); + +return 0 ; +} + +static int io_cgroup_disk_sectors_read (struct cgroup *cgroup, +struct cftype *cftype,struct seq_file *m) +{ +struct io_cgroup *iocg; +struct io_group *iog; +struct hlist_node *n; + +if (!cgroup_lock_live_group (cgroup)) +return -ENODEV ; + + iocg =cgroup_to_io_cgroup (cgroup); + +spin_lock_irq (&iocg->lock); +hlist_for_each_entry_rcu (iog, n, &iocg->group_data, group_node) { +/* +* There might be groups which are not functional and +* waiting to be reclaimed upon cgoup deletion. +*/ +if (iog->key) { +seq_printf (m,"%u %u %lu\n" ,MAJOR (iog->dev), +MINOR (iog->dev), + iog->entity.total_sector_service); + } + } +spin_unlock_irq (&iocg->lock); +cgroup_unlock (); + +return 0 ; +} +/** * bfq_group_chain_alloc - allocate a chain of groups. * @bfqd: queue descriptor.
@@ -1155,7 +1219,7 @@ STORE_FUNCTION(ioprio_class, IOPRIO_CLASS_RT, IOPRIO_CLASS_IDLE);* to the root has already an allocated group on @bfqd. */ struct io_group *io_group_chain_alloc (struct request_queue *q,void *key, -struct cgroup *cgroup) +struct cgroup *cgroup,struct bio *bio) {struct io_cgroup *iocg;struct io_group *iog, *leaf =NULL , *prev =NULL ;
@@ -1180,6 +1244,9 @@ struct io_group *io_group_chain_alloc(struct request_queue *q, void *key, iog->iocg_id =css_id (&iocg->css); +sscanf (dev_name (bdi->dev),"%u:%u" , &major, &minor); + iog->dev =MKDEV (major, minor); +io_group_init_entity (iocg, iog); iog->my_entity = &iog->entity;
@@ -1297,7 +1364,7 @@ void io_group_chain_link(struct request_queue *q, void *key,*/ struct io_group *io_find_alloc_group (struct request_queue *q,struct cgroup *cgroup,struct elv_fq_data *efqd, -int create) +int create,struct bio *bio) {struct io_cgroup *iocg =cgroup_to_io_cgroup (cgroup);struct io_group *iog =NULL ;
@@ -1316,7 +1383,7 @@ struct io_group *io_find_alloc_group(struct request_queue *q,if (iog !=NULL || !create) goto end; - iog =io_group_chain_alloc (q, key, cgroup); + iog =io_group_chain_alloc (q, key, cgroup, bio);if (iog !=NULL )io_group_chain_link (q, key, cgroup, iog, efqd);
@@ -1346,7 +1413,7 @@ struct io_group *io_get_io_group(struct request_queue *q, int create)rcu_read_lock (); cgroup =task_cgroup (current, io_subsys_id); - iog =io_find_alloc_group (q, cgroup, efqd, create); + iog =io_find_alloc_group (q, cgroup, efqd, create,NULL );if (!iog) {if (create) iog = efqd->root_group;
@@ -1421,6 +1488,14 @@ struct cftype bfqio_files[] = { .read_u64 = io_cgroup_ioprio_class_read, .write_u64 = io_cgroup_ioprio_class_write, }, + { + .name = "disk_time", + .read_seq_string = io_cgroup_disk_time_read, + }, + { + .name = "disk_sectors", + .read_seq_string = io_cgroup_disk_sectors_read, + }, };int iocg_populate (struct cgroup_subsys *subsys,struct cgroup *cgroup)
@@ -1868,7 +1943,7 @@ EXPORT_SYMBOL(elv_get_slice_idle);void elv_ioq_served (struct io_queue *ioq,bfq_service_t served) { -entity_served (&ioq->entity, served); +entity_served (&ioq->entity, served, ioq->nr_sectors); }/* Tells whether ioq is queued in root group or not */
block/elevator-fq.h +10 -0
@@ -150,6 +150,13 @@ struct io_entity {unsigned short ioprio_class, new_ioprio_class;int ioprio_changed; + +/* +* Keep track of total service received by this entity. Keep the +* stats both for time slices and number of sectors dispatched +*/ +unsigned long total_service; +unsigned long total_sector_service; };/*
@@ -243,6 +250,9 @@ struct io_group {int deleting;unsigned short iocg_id; + +/* The device MKDEV(major, minor), this group has been created for */ +dev_t dev; };/**
Signature · 1 line
1.6.0.6
PATCH 8/20 io-controller: idle for sometime on sync queue before expiring it
In reply to a message not in the archiveo When a sync queue expires, in many cases it might be empty and then it will be deleted from the active tree. This will lead to a scenario where out of two competing queues, only one is on the tree and when a new queue is selected, vtime jump takes place and we don't see services provided in proportion to weight.
o In general this is a fundamental problem with fairness of sync queues where queues are not continuously backlogged. Looks like idling is only solution to make sure such kind of queues can get some decent amount of disk bandwidth in the face of competion from continusouly backlogged queues. But excessive idling has potential to reduce performance on SSD and disks with commnad queuing.
o This patch experiments with waiting for next request to come before a queue is expired after it has consumed its time slice. This can ensure more accurate fairness numbers in some cases.
o Introduced a tunable "fairness". If set, io-controller will put more focus on getting fairness right than getting throughput right.
--- block/cfq-iosched.c | 1 + block/elevator-fq.c | 131 ++++++++++++++++++++++++++++++++++++++++++++------ block/elevator-fq.h | 15 ++++++ 3 files changed, 131 insertions(+), 16 deletions(-)
block/cfq-iosched.c +1 -0
@@ -2004,6 +2004,7 @@ static struct elv_fs_entry cfq_attrs[] = {ELV_ATTR (slice_idle),ELV_ATTR (slice_sync),ELV_ATTR (slice_async), +ELV_ATTR (fairness), __ATTR_NULL };
block/elevator-fq.c +115 -16
267 lines not on this page · raw message
block/elevator-fq.h +15 -0
@@ -77,6 +77,7 @@ struct io_service_tree {struct io_sched_data {struct io_entity *active_entity;struct io_entity *next_active; +int nr_active;struct io_service_tree service_tree[IO_IOPRIO_CLASSES ]; };
@@ -331,6 +332,13 @@ struct elv_fq_data {unsigned long long rate_sampling_start;/*sampling window start jifies*/ /* number of sectors finished io during current sampling window */ unsigned long rate_sectors_current; + +/* +* If set to 1, will disable many optimizations done for boost +* throughput and focus more on providing fairness for sync +* queues. +*/ +unsigned int fairness; };extern int elv_slice_idle;
@@ -355,6 +363,8 @@ enum elv_queue_state_flags { ELV_QUEUE_FLAG_wait_request,/* waiting for a request */ ELV_QUEUE_FLAG_must_dispatch,/* must be allowed a dispatch */ ELV_QUEUE_FLAG_slice_new,/* no requests dispatched in slice */ + ELV_QUEUE_FLAG_wait_busy,/* wait for this queue to get busy */ + ELV_QUEUE_FLAG_wait_busy_done,/* Have already waited on this queue*/ ELV_QUEUE_FLAG_NR, };
@@ -378,6 +388,8 @@ ELV_IO_QUEUE_FLAG_FNS(wait_request) ELV_IO_QUEUE_FLAG_FNS(must_dispatch) ELV_IO_QUEUE_FLAG_FNS(idle_window) ELV_IO_QUEUE_FLAG_FNS(slice_new) +ELV_IO_QUEUE_FLAG_FNS(wait_busy) +ELV_IO_QUEUE_FLAG_FNS(wait_busy_done) static inline struct io_service_tree * io_entity_service_tree(struct io_entity *entity)
@@ -550,6 +562,9 @@ extern ssize_t elv_slice_sync_store(struct elevator_queue *q, const char *name,extern ssize_t elv_slice_async_show (struct elevator_queue *q,char *name);extern ssize_t elv_slice_async_store (struct elevator_queue *q,const char *name,size_t count); +extern ssize_t elv_fairness_show (struct elevator_queue *q,char *name); +extern ssize_t elv_fairness_store (struct elevator_queue *q,const char *name, +size_t count);/* Functions used by elevator.c */ extern int elv_init_fq_data (struct request_queue *q,struct elevator_queue *e);
Signature · 1 line
1.6.0.6
PATCH 9/20 io-controller: Separate out queue and data
In reply to a message not in the archiveo So far noop, deadline and AS had one common structure called *_data which contained both the queue information where requests are queued and also common data used for scheduling. This patch breaks down this common structure in two parts, *_queue and *_data. This is along the lines of cfq where all the reuquests are queued in queue and common data and tunables are part of data.
o It does not change the functionality but this re-organization helps once noop, deadline and AS are changed to use hierarchical fair queuing.
o looks like queue_empty function is not required and we can check for q->nr_sorted in elevator layer to see if ioscheduler queues are empty or not.
--- block/as-iosched.c | 208 ++++++++++++++++++++++++++-------------------- block/deadline-iosched.c | 117 ++++++++++++++++---------- block/elevator.c | 111 +++++++++++++++++++++---- block/noop-iosched.c | 59 ++++++------- include/linux/elevator.h | 8 ++- 5 files changed, 319 insertions(+), 184 deletions(-)
block/as-iosched.c +118 -90
@@ -76,13 +76,7 @@ enum anticipation_status {* or timed out */ }; -struct as_data { -/* -* run time data -*/ - -struct request_queue *q;/* the "owner" queue */ - +struct as_queue {/* * requests (as_rq s) are present on both sort_list and fifo_list */
@@ -90,6 +84,14 @@ struct as_data {struct list_head fifo_list[2 ];struct request *next_rq[2 ];/* next in sort order */ +unsigned long last_check_fifo[2 ]; +int write_batch_count;/* max # of reqs in a write batch */ +int current_write_count;/* how many requests left this batch */ +int write_batch_idled;/* has the write batch gone idle? */ +}; + +struct as_data { +struct request_queue *q;/* the "owner" queue */ sector_t last_sector[2 ];/* last SYNC & ASYNC sectors */ unsigned long exit_prob;/* probability a task will exit while
@@ -103,21 +105,17 @@ struct as_data {sector_t new_seek_mean;unsigned long current_batch_expires; -unsigned long last_check_fifo[2 ];int changed_batch;/* 1: waiting for old batch to end */ int new_batch;/* 1: waiting on first read complete */ -int batch_data_dir;/* current batch SYNC / ASYNC */ -int write_batch_count;/* max # of reqs in a write batch */ -int current_write_count;/* how many requests left this batch */ -int write_batch_idled;/* has the write batch gone idle? */ enum anticipation_status antic_status;unsigned long antic_start;/* jiffies: when it started */ struct timer_list antic_timer;/* anticipatory scheduling timer */ -struct work_struct antic_work;/* Deferred unplugging */ +struct work_struct antic_work;/* Deferred unplugging */ struct io_context *io_context;/* Identify the expected process */ int ioc_finished;/* IO associated with io_context is finished */ int nr_dispatched; +int batch_data_dir;/* current batch SYNC / ASYNC */ /* * settings that change how the i/o scheduler behaves
@@ -258,13 +256,14 @@ static void as_put_io_context(struct request *rq)/* * rb tree support functions */ -#define RQ_RB_ROOT (ad, rq) (&(ad)->sort_list[rq_is_sync((rq))]) +#define RQ_RB_ROOT (asq, rq) (&(asq)->sort_list[rq_is_sync((rq))])static void as_add_rq_rb (struct as_data *ad,struct request *rq) {struct request *alias; +struct as_queue *asq =elv_get_sched_queue (ad->q, rq); -while ((unlikely (alias =elv_rb_add (RQ_RB_ROOT (ad, rq), rq)))) { +while ((unlikely (alias =elv_rb_add (RQ_RB_ROOT (asq, rq), rq)))) {as_move_to_dispatch (ad, alias);as_antic_stop (ad); }
@@ -272,7 +271,9 @@ static void as_add_rq_rb(struct as_data *ad, struct request *rq)static inline void as_del_rq_rb (struct as_data *ad,struct request *rq) { -elv_rb_del (RQ_RB_ROOT (ad, rq), rq); +struct as_queue *asq =elv_get_sched_queue (ad->q, rq); + +elv_rb_del (RQ_RB_ROOT (asq, rq), rq); }/*
@@ -366,7 +367,7 @@ as_choose_req(struct as_data *ad, struct request *rq1, struct request *rq2)* what request to process next. Anticipation works on top of this. */ static struct request * -as_find_next_rq (struct as_data *ad,struct request *last) +as_find_next_rq (struct as_data *ad,struct as_queue *asq,struct request *last) {struct rb_node *rbnext =rb_next (&last->rb_node);struct rb_node *rbprev =rb_prev (&last->rb_node);
@@ -382,7 +383,7 @@ as_find_next_rq(struct as_data *ad, struct request *last)else {const int data_dir =rq_is_sync (last); - rbnext =rb_first (&ad->sort_list[data_dir]); + rbnext =rb_first (&asq->sort_list[data_dir]);if (rbnext && rbnext != &last->rb_node) next =rb_entry_rq (rbnext); }
@@ -787,9 +788,10 @@ static int as_can_anticipate(struct as_data *ad, struct request *rq)static void as_update_rq (struct as_data *ad,struct request *rq) {const int data_dir =rq_is_sync (rq); +struct as_queue *asq =elv_get_sched_queue (ad->q, rq);/* keep the next_rq cache up to date */ - ad->next_rq[data_dir] =as_choose_req (ad, rq, ad->next_rq[data_dir]); + asq->next_rq[data_dir] =as_choose_req (ad, rq, asq->next_rq[data_dir]);/* * have we been anticipating this request?
@@ -810,25 +812,26 @@ static void update_write_batch(struct as_data *ad) {unsigned long batch = ad->batch_expire[BLK_RW_ASYNC ];long write_time; +struct as_queue *asq =elv_get_sched_queue (ad->q,NULL ); write_time = (jiffies - ad->current_batch_expires) + batch;if (write_time <0 ) write_time =0 ; -if (write_time > batch && !ad->write_batch_idled) { +if (write_time > batch && !asq->write_batch_idled) {if (write_time > batch *3 ) - ad->write_batch_count /=2 ; + asq->write_batch_count /=2 ;else - ad->write_batch_count--; - }else if (write_time < batch && ad->current_write_count ==0 ) { + asq->write_batch_count--; + }else if (write_time < batch && asq->current_write_count ==0 ) {if (batch > write_time *3 ) - ad->write_batch_count *=2 ; + asq->write_batch_count *=2 ;else - ad->write_batch_count++; + asq->write_batch_count++; } -if (ad->write_batch_count <1 ) - ad->write_batch_count =1 ; +if (asq->write_batch_count <1 ) + asq->write_batch_count =1 ; }/*
@@ -899,6 +902,7 @@ static void as_remove_queued_request(struct request_queue *q,const int data_dir =rq_is_sync (rq);struct as_data *ad = q->elevator->elevator_data;struct io_context *ioc; +struct as_queue *asq =elv_get_sched_queue (q, rq);WARN_ON (RQ_STATE (rq) !=AS_RQ_QUEUED );
@@ -912,8 +916,8 @@ static void as_remove_queued_request(struct request_queue *q,* Update the "next_rq" cache if we are about to remove its * entry */ -if (ad->next_rq[data_dir] == rq) - ad->next_rq[data_dir] =as_find_next_rq (ad, rq); +if (asq->next_rq[data_dir] == rq) + asq->next_rq[data_dir] =as_find_next_rq (ad, asq, rq);rq_fifo_clear (rq);as_del_rq_rb (ad, rq);
@@ -927,23 +931,23 @@ static void as_remove_queued_request(struct request_queue *q,* * See as_antic_expired comment. */ -static int as_fifo_expired (struct as_data *ad,int adir) +static int as_fifo_expired (struct as_data *ad,struct as_queue *asq,int adir) {struct request *rq;long delta_jif; - delta_jif = jiffies - ad->last_check_fifo[adir]; + delta_jif = jiffies - asq->last_check_fifo[adir];if (unlikely (delta_jif <0 )) delta_jif = -delta_jif;if (delta_jif < ad->fifo_expire[adir])return 0 ; - ad->last_check_fifo[adir] = jiffies; + asq->last_check_fifo[adir] = jiffies; -if (list_empty (&ad->fifo_list[adir])) +if (list_empty (&asq->fifo_list[adir]))return 0 ; - rq =rq_entry_fifo (ad->fifo_list[adir].next); + rq =rq_entry_fifo (asq->fifo_list[adir].next);return time_after (jiffies,rq_fifo_time (rq)); }
@@ -952,7 +956,7 @@ static int as_fifo_expired(struct as_data *ad, int adir)* as_batch_expired returns true if the current batch has expired. A batch * is a set of reads or a set of writes. */ -static inline int as_batch_expired (struct as_data *ad) +static inline int as_batch_expired (struct as_data *ad,struct as_queue *asq) {if (ad->changed_batch || ad->new_batch)return 0 ;
@@ -962,7 +966,7 @@ static inline int as_batch_expired(struct as_data *ad)return time_after (jiffies, ad->current_batch_expires);return time_after (jiffies, ad->current_batch_expires) - || ad->current_write_count ==0 ; + || asq->current_write_count ==0 ; }/*
@@ -971,6 +975,7 @@ static inline int as_batch_expired(struct as_data *ad)static void as_move_to_dispatch (struct as_data *ad,struct request *rq) {const int data_dir =rq_is_sync (rq); +struct as_queue *asq =elv_get_sched_queue (ad->q, rq);BUG_ON (RB_EMPTY_NODE (&rq->rb_node));
@@ -993,12 +998,12 @@ static void as_move_to_dispatch(struct as_data *ad, struct request *rq) ad->io_context =NULL ; } -if (ad->current_write_count !=0 ) - ad->current_write_count--; +if (asq->current_write_count !=0 ) + asq->current_write_count--; } ad->ioc_finished =0 ; - ad->next_rq[data_dir] =as_find_next_rq (ad, rq); + asq->next_rq[data_dir] =as_find_next_rq (ad, asq, rq);/* * take it off the sort and fifo list, add to dispatch queue
@@ -1022,9 +1027,16 @@ static void as_move_to_dispatch(struct as_data *ad, struct request *rq)static int as_dispatch_request (struct request_queue *q,int force) {struct as_data *ad = q->elevator->elevator_data; -const int reads = !list_empty (&ad->fifo_list[BLK_RW_SYNC ]); -const int writes = !list_empty (&ad->fifo_list[BLK_RW_ASYNC ]);struct request *rq; +struct as_queue *asq =elv_select_sched_queue (q, force); +int reads, writes; + +if (!asq) +return 0 ; + + reads = !list_empty (&asq->fifo_list[BLK_RW_SYNC ]); + writes = !list_empty (&asq->fifo_list[BLK_RW_ASYNC ]); +if (unlikely (force)) {/*
@@ -1040,25 +1052,25 @@ static int as_dispatch_request(struct request_queue *q, int force) ad->changed_batch =0 ; ad->new_batch =0 ; -while (ad->next_rq[BLK_RW_SYNC ]) { -as_move_to_dispatch (ad, ad->next_rq[BLK_RW_SYNC ]); +while (asq->next_rq[BLK_RW_SYNC ]) { +as_move_to_dispatch (ad, asq->next_rq[BLK_RW_SYNC ]); dispatched++; } - ad->last_check_fifo[BLK_RW_SYNC ] = jiffies; + asq->last_check_fifo[BLK_RW_SYNC ] = jiffies; -while (ad->next_rq[BLK_RW_ASYNC ]) { -as_move_to_dispatch (ad, ad->next_rq[BLK_RW_ASYNC ]); +while (asq->next_rq[BLK_RW_ASYNC ]) { +as_move_to_dispatch (ad, asq->next_rq[BLK_RW_ASYNC ]); dispatched++; } - ad->last_check_fifo[BLK_RW_ASYNC ] = jiffies; + asq->last_check_fifo[BLK_RW_ASYNC ] = jiffies;return dispatched; }/* Signal that the write batch was uncontended, so we can't time it */ if (ad->batch_data_dir ==BLK_RW_ASYNC && !reads) { -if (ad->current_write_count ==0 || !writes) - ad->write_batch_idled =1 ; +if (asq->current_write_count ==0 || !writes) + asq->write_batch_idled =1 ; } if (!(reads || writes)
@@ -1067,14 +1079,14 @@ static int as_dispatch_request(struct request_queue *q, int force) || ad->changed_batch)return 0 ; -if (!(reads && writes &&as_batch_expired (ad))) { +if (!(reads && writes &&as_batch_expired (ad, asq))) {/* * batch is still running or no reads or no writes */ - rq = ad->next_rq[ad->batch_data_dir]; + rq = asq->next_rq[ad->batch_data_dir];if (ad->batch_data_dir ==BLK_RW_SYNC && ad->antic_expire) { -if (as_fifo_expired (ad,BLK_RW_SYNC )) +if (as_fifo_expired (ad, asq,BLK_RW_SYNC )) goto fifo_expired;if (as_can_anticipate (ad, rq)) {
@@ -1098,7 +1110,7 @@ static int as_dispatch_request(struct request_queue *q, int force)*/ if (reads) { -BUG_ON (RB_EMPTY_ROOT (&ad->sort_list[BLK_RW_SYNC ])); +BUG_ON (RB_EMPTY_ROOT (&asq->sort_list[BLK_RW_SYNC ]));if (writes && ad->batch_data_dir ==BLK_RW_SYNC )/*
@@ -1111,8 +1123,8 @@ static int as_dispatch_request(struct request_queue *q, int force) ad->changed_batch =1 ; } ad->batch_data_dir =BLK_RW_SYNC ; - rq =rq_entry_fifo (ad->fifo_list[BLK_RW_SYNC ].next); - ad->last_check_fifo[ad->batch_data_dir] = jiffies; + rq =rq_entry_fifo (asq->fifo_list[BLK_RW_SYNC ].next); + asq->last_check_fifo[ad->batch_data_dir] = jiffies; goto dispatch_request; }
@@ -1122,7 +1134,7 @@ static int as_dispatch_request(struct request_queue *q, int force)if (writes) { dispatch_writes: -BUG_ON (RB_EMPTY_ROOT (&ad->sort_list[BLK_RW_ASYNC ])); +BUG_ON (RB_EMPTY_ROOT (&asq->sort_list[BLK_RW_ASYNC ]));if (ad->batch_data_dir ==BLK_RW_SYNC ) { ad->changed_batch =1 ;
@@ -1135,10 +1147,10 @@ dispatch_writes: ad->new_batch =0 ; } ad->batch_data_dir =BLK_RW_ASYNC ; - ad->current_write_count = ad->write_batch_count; - ad->write_batch_idled =0 ; - rq =rq_entry_fifo (ad->fifo_list[BLK_RW_ASYNC ].next); - ad->last_check_fifo[BLK_RW_ASYNC ] = jiffies; + asq->current_write_count = asq->write_batch_count; + asq->write_batch_idled =0 ; + rq =rq_entry_fifo (asq->fifo_list[BLK_RW_ASYNC ].next); + asq->last_check_fifo[BLK_RW_ASYNC ] = jiffies; goto dispatch_request; }
@@ -1150,9 +1162,9 @@ dispatch_request:* If a request has expired, service it. */ -if (as_fifo_expired (ad, ad->batch_data_dir)) { +if (as_fifo_expired (ad, asq, ad->batch_data_dir)) { fifo_expired: - rq =rq_entry_fifo (ad->fifo_list[ad->batch_data_dir].next); + rq =rq_entry_fifo (asq->fifo_list[ad->batch_data_dir].next); }if (ad->changed_batch) {
@@ -1185,6 +1197,7 @@ static void as_add_request(struct request_queue *q, struct request *rq) {struct as_data *ad = q->elevator->elevator_data;int data_dir; +struct as_queue *asq =elv_get_sched_queue (q, rq);RQ_SET_STATE (rq,AS_RQ_NEW );
@@ -1203,7 +1216,7 @@ static void as_add_request(struct request_queue *q, struct request *rq)* set expire time and add to fifo list */ rq_set_fifo_time (rq, jiffies + ad->fifo_expire[data_dir]); -list_add_tail (&rq->queuelist, &ad->fifo_list[data_dir]); +list_add_tail (&rq->queuelist, &asq->fifo_list[data_dir]);as_update_rq (ad, rq);/* keep state machine up to date */ RQ_SET_STATE (rq,AS_RQ_QUEUED );
@@ -1225,31 +1238,20 @@ static void as_deactivate_request(struct request_queue *q, struct request *rq)atomic_inc (&RQ_IOC (rq)->aic->nr_dispatched); } -/* -* as_queue_empty tells us if there are requests left in the device. It may -* not be the case that a driver can get the next request even if the queue -* is not empty - it is used in the block layer to check for plugging and -* merging opportunities -*/ -static int as_queue_empty (struct request_queue *q) -{ -struct as_data *ad = q->elevator->elevator_data; - -return list_empty (&ad->fifo_list[BLK_RW_ASYNC ]) - &&list_empty (&ad->fifo_list[BLK_RW_SYNC ]); -} -static int as_merge (struct request_queue *q,struct request **req,struct bio *bio) { -struct as_data *ad = q->elevator->elevator_data;sector_t rb_key = bio->bi_sector +bio_sectors (bio);struct request *__rq; +struct as_queue *asq =elv_get_sched_queue_current (q); + +if (!asq) +return ELEVATOR_NO_MERGE ;/* * check for front merge */ - __rq =elv_rb_find (&ad->sort_list[bio_data_dir (bio)], rb_key); + __rq =elv_rb_find (&asq->sort_list[bio_data_dir (bio)], rb_key);if (__rq &&elv_rq_merge_ok (__rq, bio)) { *req = __rq;return ELEVATOR_FRONT_MERGE ;
@@ -1336,6 +1338,41 @@ static int as_may_queue(struct request_queue *q, int rw)return ret; } +/* Called with queue lock held */ +static void *as_alloc_as_queue (struct request_queue *q, +struct elevator_queue *eq,gfp_t gfp_mask) +{ +struct as_queue *asq; +struct as_data *ad = eq->elevator_data; + + asq =kmalloc_node (sizeof (*asq), gfp_mask | __GFP_ZERO, q->node); +if (asq ==NULL ) + goto out; + +INIT_LIST_HEAD (&asq->fifo_list[BLK_RW_SYNC ]); +INIT_LIST_HEAD (&asq->fifo_list[BLK_RW_ASYNC ]); + asq->sort_list[BLK_RW_SYNC ] =RB_ROOT ; + asq->sort_list[BLK_RW_ASYNC ] =RB_ROOT ; +if (ad) + asq->write_batch_count = ad->batch_expire[BLK_RW_ASYNC ] /10 ; +else + asq->write_batch_count = default_write_batch_expire /10 ; + +if (asq->write_batch_count <2 ) + asq->write_batch_count =2 ; +out: +return asq; +} + +static void as_free_as_queue (struct elevator_queue *e,void *sched_queue) +{ +struct as_queue *asq = sched_queue; + +BUG_ON (!list_empty (&asq->fifo_list[BLK_RW_SYNC ])); +BUG_ON (!list_empty (&asq->fifo_list[BLK_RW_ASYNC ])); +kfree (asq); +} +static void as_exit_queue (struct elevator_queue *e) {struct as_data *ad = e->elevator_data;
@@ -1343,9 +1380,6 @@ static void as_exit_queue(struct elevator_queue *e)del_timer_sync (&ad->antic_timer);cancel_work_sync (&ad->antic_work); -BUG_ON (!list_empty (&ad->fifo_list[BLK_RW_SYNC ])); -BUG_ON (!list_empty (&ad->fifo_list[BLK_RW_ASYNC ])); -put_io_context (ad->io_context);kfree (ad); }
@@ -1369,10 +1403,6 @@ static void *as_init_queue(struct request_queue *q)init_timer (&ad->antic_timer);INIT_WORK (&ad->antic_work, as_work_handler); -INIT_LIST_HEAD (&ad->fifo_list[BLK_RW_SYNC ]); -INIT_LIST_HEAD (&ad->fifo_list[BLK_RW_ASYNC ]); - ad->sort_list[BLK_RW_SYNC ] =RB_ROOT ; - ad->sort_list[BLK_RW_ASYNC ] =RB_ROOT ; ad->fifo_expire[BLK_RW_SYNC ] = default_read_expire; ad->fifo_expire[BLK_RW_ASYNC ] = default_write_expire; ad->antic_expire = default_antic_expire;
@@ -1380,9 +1410,6 @@ static void *as_init_queue(struct request_queue *q) ad->batch_expire[BLK_RW_ASYNC ] = default_write_batch_expire; ad->current_batch_expires = jiffies + ad->batch_expire[BLK_RW_SYNC ]; - ad->write_batch_count = ad->batch_expire[BLK_RW_ASYNC ] /10 ; -if (ad->write_batch_count <2 ) - ad->write_batch_count =2 ;return ad; }
@@ -1480,7 +1507,6 @@ static struct elevator_type iosched_as = { .elevator_add_req_fn = as_add_request, .elevator_activate_req_fn = as_activate_request, .elevator_deactivate_req_fn = as_deactivate_request, - .elevator_queue_empty_fn = as_queue_empty, .elevator_completed_req_fn = as_completed_request, .elevator_former_req_fn = elv_rb_former_request, .elevator_latter_req_fn = elv_rb_latter_request,
@@ -1488,6 +1514,8 @@ static struct elevator_type iosched_as = { .elevator_init_fn = as_init_queue, .elevator_exit_fn = as_exit_queue, .trim = as_trim, + .elevator_alloc_sched_queue_fn = as_alloc_as_queue, + .elevator_free_sched_queue_fn = as_free_as_queue, }, .elevator_attrs = as_attrs,
block/deadline-iosched.c +71 -46
266 lines not on this page · raw message
block/elevator.c +96 -15
179 lines not on this page · raw message
block/noop-iosched.c +28 -31
111 lines not on this page · raw message
include/linux/elevator.h +6 -2
@@ -30,8 +30,9 @@ typedef void (elevator_deactivate_req_fn) (struct request_queue *, struct requestypedef void *(elevator_init_fn ) (struct request_queue *);typedef void (elevator_exit_fn ) (struct elevator_queue *); -#ifdef CONFIG_ELV_FAIR_QUEUING +typedef void * (elevator_alloc_sched_queue_fn ) (struct request_queue *q,struct elevator_queue *eq,gfp_t );typedef void (elevator_free_sched_queue_fn ) (struct elevator_queue *,void *); +#ifdef CONFIG_ELV_FAIR_QUEUING typedef void (elevator_active_ioq_set_fn ) (struct request_queue *,void *,int );typedef void (elevator_active_ioq_reset_fn ) (struct request_queue *,void *);typedef void (elevator_arm_slice_timer_fn ) (struct request_queue *,void *);
@@ -70,8 +71,9 @@ struct elevator_opselevator_exit_fn *elevator_exit_fn;void (*trim)(struct io_context *); -#ifdef CONFIG_ELV_FAIR_QUEUING +elevator_alloc_sched_queue_fn *elevator_alloc_sched_queue_fn;elevator_free_sched_queue_fn *elevator_free_sched_queue_fn; +#ifdef CONFIG_ELV_FAIR_QUEUING elevator_active_ioq_set_fn *elevator_active_ioq_set_fn;elevator_active_ioq_reset_fn *elevator_active_ioq_reset_fn;
@@ -112,6 +114,7 @@ struct elevator_queue {struct elevator_ops *ops;void *elevator_data; +void *sched_queue;struct kobject kobj;struct elevator_type *elevator_type;struct mutex sysfs_lock;
@@ -260,5 +263,6 @@ static inline int elv_iosched_fair_queuing_enabled(struct elevator_queue *e)#endif /* ELV_IOSCHED_FAIR_QUEUING */ extern void *elv_get_sched_queue (struct request_queue *q,struct request *rq);extern void *elv_select_sched_queue (struct request_queue *q,int force); +extern void *elv_get_sched_queue_current (struct request_queue *q);#endif /* CONFIG_BLOCK */ #endif
Signature · 1 line
1.6.0.6
PATCH 10/20 io-conroller: Prepare elevator layer for single queue schedulers
In reply to a message not in the archiveElevator layer now has support for hierarchical fair queuing. cfq has been migrated to make use of it and now it is time to do groundwork for noop, deadline and AS.
noop deadline and AS don't maintain separate queues for different processes. There is only one single queue. Effectively one can think that in hierarchical setup, there will be one queue per cgroup where requests from all the processes in the cgroup will be queued.
Generally io scheduler takes care of creating queues. Because there is only one queue here, we have modified common layer to take care of queue creation and some other functionality. This special casing helps in keeping the changes to noop, deadline and AS to the minimum.
--- block/as-iosched.c | 2 +- block/deadline-iosched.c | 2 +- block/elevator-fq.c | 206 +++++++++++++++++++++++++++++++++++++++++++++- block/elevator-fq.h | 70 ++++++++++++++++ block/elevator.c | 37 ++++++++- block/noop-iosched.c | 2 +- include/linux/elevator.h | 16 ++++- 7 files changed, 327 insertions(+), 8 deletions(-)
block/as-iosched.c +1 -1
@@ -1340,7 +1340,7 @@ static int as_may_queue(struct request_queue *q, int rw)/* Called with queue lock held */ static void *as_alloc_as_queue (struct request_queue *q, -struct elevator_queue *eq,gfp_t gfp_mask) +struct elevator_queue *eq,gfp_t gfp_mask,struct io_queue *ioq) {struct as_queue *asq;struct as_data *ad = eq->elevator_data;
block/deadline-iosched.c +1 -1
@@ -341,7 +341,7 @@ dispatch_request: }static void *deadline_alloc_deadline_queue (struct request_queue *q, -struct elevator_queue *eq,gfp_t gfp_mask) +struct elevator_queue *eq,gfp_t gfp_mask,struct io_queue *ioq) {struct deadline_queue *dq;
block/elevator-fq.c +204 -2
260 lines not on this page · raw message
block/elevator-fq.h +70 -0
105 lines not on this page · raw message
block/elevator.c +35 -2
@@ -211,9 +211,17 @@ static void *elevator_alloc_sched_queue(struct request_queue *q, {void *sched_queue =NULL ; +/* +* If fair queuing is enabled, then queue allocation takes place +* during set_request() functions when request actually comes +* in. +*/ +if (elv_iosched_fair_queuing_enabled (eq)) +return NULL ; +if (eq->ops->elevator_alloc_sched_queue_fn) { sched_queue = eq->ops->elevator_alloc_sched_queue_fn (q, eq, -GFP_KERNEL ); +GFP_KERNEL ,NULL );if (!sched_queue)return ERR_PTR (-ENOMEM );
@@ -963,6 +971,13 @@ int elv_set_request(struct request_queue *q, struct request *rq, gfp_t gfp_mask) {struct elevator_queue *e = q->elevator; +/* +* Optimization for noop, deadline and AS which maintain only single +* ioq per io group +*/ +if (elv_iosched_single_ioq (e)) +return elv_fq_set_request_ioq (q, rq, gfp_mask); +if (e->ops->elevator_set_req_fn)return e->ops->elevator_set_req_fn (q, rq, gfp_mask);
@@ -974,6 +989,15 @@ void elv_put_request(struct request_queue *q, struct request *rq) {struct elevator_queue *e = q->elevator; +/* +* Optimization for noop, deadline and AS which maintain only single +* ioq per io group +*/ +if (elv_iosched_single_ioq (e)) { +elv_fq_unset_request_ioq (q, rq); +return ; + } +if (e->ops->elevator_put_req_fn) e->ops->elevator_put_req_fn (rq); }
@@ -1345,9 +1369,18 @@ EXPORT_SYMBOL(elv_select_sched_queue);/* * Get the io scheduler queue pointer for current task. +* +* If fair queuing is enabled, determine the io group of task and retrieve +* the ioq pointer from that. This is used by only single queue ioschedulers +* for retrieving the queue associated with the group to decide whether the +* new bio can do a front merge or not. */ void *elv_get_sched_queue_current (struct request_queue *q) { -return q->elevator->sched_queue; +/* Fair queuing is not enabled. There is only one queue. */ +if (!elv_iosched_fair_queuing_enabled (q->elevator)) +return q->elevator->sched_queue; + +return ioq_sched_queue (elv_lookup_ioq_current (q)); }EXPORT_SYMBOL (elv_get_sched_queue_current);
block/noop-iosched.c +1 -1
@@ -62,7 +62,7 @@ noop_latter_request(struct request_queue *q, struct request *rq) }static void *noop_alloc_noop_queue (struct request_queue *q, -struct elevator_queue *eq,gfp_t gfp_mask) +struct elevator_queue *eq,gfp_t gfp_mask,struct io_queue *ioq) {struct noop_queue *nq;
include/linux/elevator.h +15 -1
@@ -30,7 +30,7 @@ typedef void (elevator_deactivate_req_fn) (struct request_queue *, struct requestypedef void *(elevator_init_fn ) (struct request_queue *);typedef void (elevator_exit_fn ) (struct elevator_queue *); -typedef void * (elevator_alloc_sched_queue_fn ) (struct request_queue *q,struct elevator_queue *eq,gfp_t ); +typedef void * (elevator_alloc_sched_queue_fn ) (struct request_queue *q,struct elevator_queue *eq,gfp_t ,struct io_queue *ioq);typedef void (elevator_free_sched_queue_fn ) (struct elevator_queue *,void *);#ifdef CONFIG_ELV_FAIR_QUEUING typedef void (elevator_active_ioq_set_fn ) (struct request_queue *,void *,int );
@@ -249,17 +249,31 @@ enum {/* iosched wants to use fq logic of elevator layer */ #define ELV_IOSCHED_NEED_FQ 1 +/* iosched maintains only single ioq per group.*/ +#define ELV_IOSCHED_SINGLE_IOQ 2 +static inline int elv_iosched_fair_queuing_enabled (struct elevator_queue *e) {return (e->elevator_type->elevator_features) &ELV_IOSCHED_NEED_FQ ; } +static inline int elv_iosched_single_ioq (struct elevator_queue *e) +{ +return (e->elevator_type->elevator_features) &ELV_IOSCHED_SINGLE_IOQ ; +} +#else /* ELV_IOSCHED_FAIR_QUEUING */ static inline int elv_iosched_fair_queuing_enabled (struct elevator_queue *e) {return 0 ; } + +static inline int elv_iosched_single_ioq (struct elevator_queue *e) +{ +return 0 ; +} +#endif /* ELV_IOSCHED_FAIR_QUEUING */ extern void *elv_get_sched_queue (struct request_queue *q,struct request *rq);extern void *elv_select_sched_queue (struct request_queue *q,int force);
Signature · 1 line
1.6.0.6
PATCH 11/20 io-controller: noop changes for hierarchical fair queuing
In reply to a message not in the archiveThis patch changes noop to use queue scheduling code from elevator layer. One can go back to old noop by deselecting CONFIG_IOSCHED_NOOP_HIER.
--- block/Kconfig.iosched | 11 +++++++++++ block/noop-iosched.c | 13 +++++++++++++ 2 files changed, 24 insertions(+), 0 deletions(-)
block/Kconfig.iosched +11 -0
@@ -25,6 +25,17 @@ config IOSCHED_NOOP that do their own scheduling and require only minimal assistance from the kernel. +config IOSCHED_NOOP_HIER + bool "Noop Hierarchical Scheduling support" + depends on IOSCHED_NOOP && CGROUPS + select ELV_FAIR_QUEUING + select GROUP_IOSCHED + default n + ---help--- + Enable hierarhical scheduling in noop. In this mode noop keeps + one IO queue per cgroup instead of a global queue. Elevator + fair queuing logic ensures fairness among various queues. + config IOSCHED_AS tristate "Anticipatory I/O scheduler" default y
block/noop-iosched.c +13 -0
@@ -82,6 +82,15 @@ static void noop_free_noop_queue(struct elevator_queue *e, void *sched_queue)kfree (nq); } +#ifdef CONFIG_IOSCHED_NOOP_HIER +static struct elv_fs_entry noop_attrs[] = { +ELV_ATTR (fairness), +ELV_ATTR (slice_idle), +ELV_ATTR (slice_sync), + __ATTR_NULL +}; +#endif +static struct elevator_type elevator_noop = { .ops = { .elevator_merge_req_fn = noop_merged_requests,
@@ -92,6 +101,10 @@ static struct elevator_type elevator_noop = { .elevator_alloc_sched_queue_fn = noop_alloc_noop_queue, .elevator_free_sched_queue_fn = noop_free_noop_queue, }, +#ifdef CONFIG_IOSCHED_NOOP_HIER + .elevator_features = ELV_IOSCHED_NEED_FQ | ELV_IOSCHED_SINGLE_IOQ, + .elevator_attrs = noop_attrs, +#endif .elevator_name = "noop", .elevator_owner = THIS_MODULE, };
Signature · 1 line
1.6.0.6
PATCH 12/20 io-controller: deadline changes for hierarchical fair queuing
In reply to a message not in the archiveThis patch changes deadline to use queue scheduling code from elevator layer. One can go back to old deadline by selecting CONFIG_IOSCHED_DEADLINE_HIER.
--- block/Kconfig.iosched | 11 +++++++++++ block/deadline-iosched.c | 8 ++++++++ 2 files changed, 19 insertions(+), 0 deletions(-)
block/Kconfig.iosched +11 -0
@@ -55,6 +55,17 @@ config IOSCHED_DEADLINE a disk at any one time, its behaviour is almost identical to the anticipatory I/O scheduler and so is a good choice. +config IOSCHED_DEADLINE_HIER + bool "Deadline Hierarchical Scheduling support" + depends on IOSCHED_DEADLINE && CGROUPS + select ELV_FAIR_QUEUING + select GROUP_IOSCHED + default n + ---help--- + Enable hierarhical scheduling in deadline. In this mode deadline keeps + one IO queue per cgroup instead of a global queue. Elevator + fair queuing logic ensures fairness among various queues. + config IOSCHED_CFQ tristate "CFQ I/O scheduler" select ELV_FAIR_QUEUING
block/deadline-iosched.c +8 -0
@@ -460,6 +460,11 @@ static struct elv_fs_entry deadline_attrs[] = {DD_ATTR (writes_starved),DD_ATTR (front_merges),DD_ATTR (fifo_batch), +#ifdef CONFIG_IOSCHED_DEADLINE_HIER +ELV_ATTR (fairness), +ELV_ATTR (slice_idle), +ELV_ATTR (slice_sync), +#endif __ATTR_NULL };
@@ -477,6 +482,9 @@ static struct elevator_type iosched_deadline = { .elevator_alloc_sched_queue_fn = deadline_alloc_deadline_queue, .elevator_free_sched_queue_fn = deadline_free_deadline_queue, }, +#ifdef CONFIG_IOSCHED_DEADLINE_HIER + .elevator_features = ELV_IOSCHED_NEED_FQ | ELV_IOSCHED_SINGLE_IOQ, +#endif .elevator_attrs = deadline_attrs, .elevator_name = "deadline", .elevator_owner = THIS_MODULE,
Signature · 1 line
1.6.0.6
PATCH 13/20 io-controller: anticipatory changes for hierarchical fair queuing
In reply to a message not in the archiveThis patch changes anticipatory scheduler to use queue scheduling code from elevator layer. One can go back to old as by deselecting CONFIG_IOSCHED_AS_HIER. Even with CONFIG_IOSCHED_AS_HIER=y, with-out any other cgroup created, AS behavior should remain the same as old.
o AS is a single queue ioschduler, that means there is one AS queue per group.
o common layer code select the queue to dispatch from based on fairness, and then AS code selects the request with-in group.
o AS runs reads and writes batches with-in group. So common layer runs timed group queues and with-in group time, AS runs timed batches of reads and writes.
o Note: Previously AS write batch length was adjusted synamically whenever a W->R batch data direction took place and when first request from the read batch completed.
Now write batch updation takes place when last request from the write batch has finished during W->R transition.
o AS runs its own anticipation logic to anticipate on reads. common layer also does the anticipation on the group if think time of the group is with-in slice_idle.
o Introduced few debugging messages in AS.
--- block/Kconfig.iosched | 12 ++ block/as-iosched.c | 280 +++++++++++++++++++++++++++++++++++++++++++++- block/elevator-fq.c | 86 ++++++++++++-- include/linux/elevator.h | 2 + 4 files changed, 363 insertions(+), 17 deletions(-)
block/Kconfig.iosched +12 -0
@@ -45,6 +45,18 @@ config IOSCHED_AS deadline I/O scheduler, it can also be slower in some cases especially some database loads. +config IOSCHED_AS_HIER + bool "Anticipatory Hierarchical Scheduling support" + depends on IOSCHED_AS && CGROUPS + select ELV_FAIR_QUEUING + select GROUP_IOSCHED + default n + ---help--- + Enable hierarhical scheduling in anticipatory. In this mode + anticipatory keeps one IO queue per cgroup instead of a global + queue. Elevator fair queuing logic ensures fairness among various + queues. + config IOSCHED_DEADLINE tristate "Deadline I/O scheduler" default y
block/as-iosched.c +274 -6
470 lines not on this page · raw message
block/elevator-fq.c +75 -11
149 lines not on this page · raw message
include/linux/elevator.h +2 -0
@@ -42,6 +42,7 @@ typedef int (elevator_update_idle_window_fn) (struct elevator_queue*, void*, struct request*);typedef struct io_queue * (elevator_close_cooperator_fn ) (struct request_queue *,void *,int probe); +typedef int (elevator_expire_ioq_fn ) (struct request_queue *,void *,int ,int );#endif struct elevator_ops
@@ -81,6 +82,7 @@ struct elevator_opselevator_should_preempt_fn *elevator_should_preempt_fn;elevator_update_idle_window_fn *elevator_update_idle_window_fn;elevator_close_cooperator_fn *elevator_close_cooperator_fn; +elevator_expire_ioq_fn *elevator_expire_ioq_fn;#endif };
Signature · 1 line
1.6.0.6
PATCH 14/20 blkio_cgroup patches from Ryo to track async bios.
In reply to a message not in the archiveo blkio_cgroup patches from Ryo to track async bios.
o Fernando is also working on another IO tracking mechanism. We are not particular about any IO tracking mechanism. This patchset can make use of any mechanism which makes it to upstream. For the time being making use of Ryo's posting.
Based on 2.6.30-rc3-git3
--- block/blk-ioc.c | 37 +++--- fs/buffer.c | 2 + fs/direct-io.c | 2 + include/linux/biotrack.h | 97 +++++++++++++ include/linux/cgroup_subsys.h | 6 + include/linux/iocontext.h | 1 + include/linux/memcontrol.h | 6 + include/linux/mmzone.h | 4 +- include/linux/page_cgroup.h | 31 ++++- init/Kconfig | 15 ++ mm/Makefile | 4 +- mm/biotrack.c | 300 +++++++++++++++++++++++++++++++++++++++++ mm/bounce.c | 2 + mm/filemap.c | 2 + mm/memcontrol.c | 6 + mm/memory.c | 5 + mm/page-writeback.c | 2 + mm/page_cgroup.c | 17 ++- mm/swap_state.c | 2 + 19 files changed, 511 insertions(+), 30 deletions(-) create mode 100644 include/linux/biotrack.h create mode 100644 mm/biotrack.c
block/blk-ioc.c +21 -16
@@ -84,27 +84,32 @@ void exit_io_context(void) } } +void init_io_context (struct io_context *ioc) +{ +atomic_set (&ioc->refcount,1 ); +atomic_set (&ioc->nr_tasks,1 ); +spin_lock_init (&ioc->lock); + ioc->ioprio_changed =0 ; + ioc->ioprio =0 ; +#ifdef CONFIG_GROUP_IOSCHED + ioc->cgroup_changed =0 ; +#endif + ioc->last_waited = jiffies;/* doesn't matter... */ + ioc->nr_batch_requests =0 ;/* because this is 0 */ + ioc->aic =NULL ; +INIT_RADIX_TREE (&ioc->radix_root,GFP_ATOMIC | __GFP_HIGH); +INIT_HLIST_HEAD (&ioc->cic_list); + ioc->ioc_data =NULL ; +} + +struct io_context *alloc_io_context (gfp_t gfp_flags,int node) {struct io_context *ret; ret =kmem_cache_alloc_node (iocontext_cachep, gfp_flags, node); -if (ret) { -atomic_set (&ret->refcount,1 ); -atomic_set (&ret->nr_tasks,1 ); -spin_lock_init (&ret->lock); - ret->ioprio_changed =0 ; - ret->ioprio =0 ; -#ifdef CONFIG_GROUP_IOSCHED - ret->cgroup_changed =0 ; -#endif - ret->last_waited = jiffies;/* doesn't matter... */ - ret->nr_batch_requests =0 ;/* because this is 0 */ - ret->aic =NULL ; -INIT_RADIX_TREE (&ret->radix_root,GFP_ATOMIC | __GFP_HIGH); -INIT_HLIST_HEAD (&ret->cic_list); - ret->ioc_data =NULL ; - } +if (ret) +init_io_context (ret);return ret; }
fs/buffer.c +2 -0
@@ -36,6 +36,7 @@#include <linux/buffer_head.h> #include <linux/task_io_accounting_ops.h> #include <linux/bio.h> +#include <linux/biotrack.h> #include <linux/notifier.h> #include <linux/cpu.h> #include <linux/bitops.h>
@@ -668,6 +669,7 @@ static void __set_page_dirty(struct page *page,if (page->mapping) {/* Race with truncate? */ WARN_ON_ONCE (warn && !PageUptodate (page));account_page_dirtied (page, mapping); +blkio_cgroup_reset_owner_pagedirty (page, current->mm);radix_tree_tag_set (&mapping->page_tree,page_index (page),PAGECACHE_TAG_DIRTY ); }
fs/direct-io.c +2 -0
@@ -33,6 +33,7 @@#include <linux/err.h> #include <linux/blkdev.h> #include <linux/buffer_head.h> +#include <linux/biotrack.h> #include <linux/rwsem.h> #include <linux/uio.h> #include <asm/atomic.h>
@@ -797,6 +798,7 @@ static int do_direct_IO(struct dio *dio) ret =PTR_ERR (page); goto out; } +blkio_cgroup_reset_owner (page, current->mm);while (block_in_page < blocks_per_page) {unsigned offset_in_page = block_in_page << blkbits;
include/linux/biotrack.h new file +97 -0
97 lines not on this page · raw message
include/linux/cgroup_subsys.h +6 -0
@@ -43,6 +43,12 @@ SUBSYS(mem_cgroup)/* */ +#ifdef CONFIG_CGROUP_BLKIO +SUBSYS(blkio_cgroup) +#endif + +/* */ +#ifdef CONFIG_CGROUP_DEVICE SUBSYS(devices)#endif
include/linux/iocontext.h +1 -0
@@ -104,6 +104,7 @@ int put_io_context(struct io_context *ioc);void exit_io_context (void );struct io_context *get_io_context (gfp_t gfp_flags,int node);struct io_context *alloc_io_context (gfp_t gfp_flags,int node); +void init_io_context (struct io_context *ioc);void copy_io_context (struct io_context **pdst,struct io_context **psrc);#else static inline void exit_io_context (void )
include/linux/memcontrol.h +6 -0
@@ -37,6 +37,8 @@ struct mm_struct;* (Of course, if memcg does memory allocation in future, GFP_KERNEL is sane.) */ +extern void __init_mem_page_cgroup (struct page_cgroup *pc); +extern int mem_cgroup_newpage_charge (struct page *page,struct mm_struct *mm,gfp_t gfp_mask);/* for swap handling */
@@ -120,6 +122,10 @@ extern bool mem_cgroup_oom_called(struct task_struct *task);#else /* CONFIG_CGROUP_MEM_RES_CTLR */ struct mem_cgroup ; +static inline void __init_mem_page_cgroup (struct page_cgroup *pc) +{ +} +static inline int mem_cgroup_newpage_charge (struct page *page,struct mm_struct *mm,gfp_t gfp_mask) {
include/linux/mmzone.h +2 -2
@@ -607,7 +607,7 @@ typedef struct pglist_data {int nr_zones;#ifdef CONFIG_FLAT_NODE_MEM_MAP /* means !SPARSEMEM */ struct page *node_mem_map; -#ifdef CONFIG_CGROUP_MEM_RES_CTLR +#ifdef CONFIG_CGROUP_PAGE struct page_cgroup *node_page_cgroup;#endif #endif
@@ -958,7 +958,7 @@ struct mem_section {/* See declaration of similar field in struct zone */ unsigned long *pageblock_flags; -#ifdef CONFIG_CGROUP_MEM_RES_CTLR +#ifdef CONFIG_CGROUP_PAGE /* * If !SPARSEMEM, pgdat doesn't have page_cgroup pointer. We use * section. (see memcontrol.h/page_cgroup.h about this.)
include/linux/page_cgroup.h +28 -3
@@ -1,7 +1,7 @@#ifndef __LINUX_PAGE_CGROUP_H#define __LINUX_PAGE_CGROUP_H -#ifdef CONFIG_CGROUP_MEM_RES_CTLR +#ifdef CONFIG_CGROUP_PAGE #include <linux/bit_spinlock.h> /* * Page Cgroup can be considered as an extended mem_map.
@@ -12,9 +12,11 @@*/ struct page_cgroup {unsigned long flags; -struct mem_cgroup *mem_cgroup;struct page *page; +#ifdef CONFIG_CGROUP_MEM_RES_CTLR +struct mem_cgroup *mem_cgroup;struct list_head lru;/* per cgroup LRU list */ +#endif };void __meminit pgdat_page_cgroup_init (struct pglist_data *pgdat);
@@ -71,7 +73,7 @@ static inline void unlock_page_cgroup(struct page_cgroup *pc) bit_spin_unlock(PCG_LOCK, &pc->flags); } -#else /* CONFIG_CGROUP_MEM_RES_CTLR */ +#else /* CONFIG_CGROUP_PAGE */ struct page_cgroup ;static inline void __meminit pgdat_page_cgroup_init (struct pglist_data *pgdat)
@@ -122,4 +124,27 @@ static inline void swap_cgroup_swapoff(int type) }#endif + +#ifdef CONFIG_CGROUP_BLKIO +/* +* use lower 16 bits for flags and reserve the rest for the page tracking id +*/ +#define PCG_TRACKING_ID_SHIFT (16) +#define PCG_TRACKING_ID_BITS \ + (8 * sizeof(unsigned long) - PCG_TRACKING_ID_SHIFT) + +/* NOTE: must be called with page_cgroup() held */ +static inline unsigned long page_cgroup_get_id (struct page_cgroup *pc) +{ +return pc->flags >>PCG_TRACKING_ID_SHIFT ; +} + +/* NOTE: must be called with page_cgroup() held */ +static inline void page_cgroup_set_id (struct page_cgroup *pc,unsigned long id) +{ +WARN_ON (id >= (1UL <<PCG_TRACKING_ID_BITS )); + pc->flags &= (1UL <<PCG_TRACKING_ID_SHIFT ) -1 ; + pc->flags |= (unsigned long )(id <<PCG_TRACKING_ID_SHIFT ); +} +#endif #endif
init/Kconfig +15 -0
@@ -616,6 +616,21 @@ config GROUP_IOSCHED endif # CGROUPS +config CGROUP_BLKIO + bool "Block I/O cgroup subsystem" + depends on CGROUPS && BLOCK + select MM_OWNER + help + Provides a Resource Controller which enables to track the onwner + of every Block I/O requests. + The information this subsystem provides can be used from any + kind of module such as dm-ioband device mapper modules or + the cfq-scheduler. + +config CGROUP_PAGE + def_bool y + depends on CGROUP_MEM_RES_CTLR || CGROUP_BLKIO + config MM_OWNER bool
mm/Makefile +3 -1
@@ -37,4 +37,6 @@ else obj-$(CONFIG_SMP) += allocpercpu.o endif obj-$(CONFIG_QUICKLIST) += quicklist.o -obj-$(CONFIG_CGROUP_MEM_RES_CTLR) += memcontrol.o page_cgroup.o +obj-$(CONFIG_CGROUP_MEM_RES_CTLR) += memcontrol.o +obj-$(CONFIG_CGROUP_PAGE) += page_cgroup.o +obj-$(CONFIG_CGROUP_BLKIO) += biotrack.o
mm/biotrack.c new file +300 -0
300 lines not on this page · raw message
mm/bounce.c +2 -0
@@ -14,6 +14,7 @@#include <linux/hash.h> #include <linux/highmem.h> #include <linux/blktrace_api.h> +#include <linux/biotrack.h> #include <trace/block.h> #include <asm/tlbflush.h>
@@ -212,6 +213,7 @@ static void __blk_queue_bounce(struct request_queue *q, struct bio **bio_orig, to->bv_len = from->bv_len; to->bv_offset = from->bv_offset;inc_zone_page_state (to->bv_page,NR_BOUNCE ); +blkio_cgroup_copy_owner (to->bv_page, page);if (rw ==WRITE ) {char *vto, *vfrom;
mm/filemap.c +2 -0
@@ -33,6 +33,7 @@#include <linux/cpuset.h> #include <linux/hardirq.h> /* for BUG_ON(!in_atomic()) only */ #include <linux/memcontrol.h> +#include <linux/biotrack.h> #include <linux/mm_inline.h> /* for page_is_file_cache() */ #include "internal.h"
@@ -464,6 +465,7 @@ int add_to_page_cache_locked(struct page *page, struct address_space *mapping, gfp_mask & GFP_RECLAIM_MASK);if (error) goto out; +blkio_cgroup_set_owner (page, current->mm); error =radix_tree_preload (gfp_mask & ~__GFP_HIGHMEM);if (error ==0 ) {
mm/memcontrol.c +6 -0
@@ -128,6 +128,12 @@ struct mem_cgroup_lru_info {struct mem_cgroup_per_node *nodeinfo[MAX_NUMNODES ]; }; +void __meminit __init_mem_page_cgroup (struct page_cgroup *pc) +{ + pc->mem_cgroup =NULL ; +INIT_LIST_HEAD (&pc->lru); +} +/* * The memory controller data structure. The memory controller controls both * page cache and RSS per cgroup. We would eventually like to provide
mm/memory.c +5 -0
@@ -51,6 +51,7 @@#include <linux/init.h> #include <linux/writeback.h> #include <linux/memcontrol.h> +#include <linux/biotrack.h> #include <linux/mmu_notifier.h> #include <linux/kallsyms.h> #include <linux/swapops.h>
@@ -2064,6 +2065,7 @@ gotten:*/ ptep_clear_flush_notify (vma, address, page_table);page_add_new_anon_rmap (new_page, vma, address); +blkio_cgroup_set_owner (new_page, mm);set_pte_at (mm, address, page_table, entry);update_mmu_cache (vma, address, entry);if (old_page) {
@@ -2529,6 +2531,7 @@ static int do_swap_page(struct mm_struct *mm, struct vm_area_struct *vma,flush_icache_page (vma, page);set_pte_at (mm, address, page_table, pte);page_add_anon_rmap (page, vma, address); +blkio_cgroup_reset_owner (page, mm);/* It's better to call commit-charge after rmap is established */ mem_cgroup_commit_charge_swapin (page, ptr);
@@ -2593,6 +2596,7 @@ static int do_anonymous_page(struct mm_struct *mm, struct vm_area_struct *vma, goto release;inc_mm_counter (mm, anon_rss);page_add_new_anon_rmap (page, vma, address); +blkio_cgroup_set_owner (page, mm);set_pte_at (mm, address, page_table, entry);/* No need to invalidate - it was non-present before */
@@ -2740,6 +2744,7 @@ static int __do_fault(struct mm_struct *mm, struct vm_area_struct *vma,if (anon) {inc_mm_counter (mm, anon_rss);page_add_new_anon_rmap (page, vma, address); +blkio_cgroup_set_owner (page, mm); }else {inc_mm_counter (mm, file_rss);page_add_file_rmap (page);
mm/page-writeback.c +2 -0
@@ -23,6 +23,7 @@#include <linux/init.h> #include <linux/backing-dev.h> #include <linux/task_io_accounting_ops.h> +#include <linux/biotrack.h> #include <linux/blkdev.h> #include <linux/mpage.h> #include <linux/rmap.h>
@@ -1243,6 +1244,7 @@ int __set_page_dirty_nobuffers(struct page *page)BUG_ON (mapping2 != mapping);WARN_ON_ONCE (!PagePrivate (page) && !PageUptodate (page));account_page_dirtied (page, mapping); +blkio_cgroup_reset_owner_pagedirty (page, current->mm);radix_tree_tag_set (&mapping->page_tree,page_index (page),PAGECACHE_TAG_DIRTY ); }
mm/page_cgroup.c +9 -8
@@ -9,14 +9,15 @@#include <linux/vmalloc.h> #include <linux/cgroup.h> #include <linux/swapops.h> +#include <linux/biotrack.h> static void __meminit __init_page_cgroup (struct page_cgroup *pc,unsigned long pfn) { pc->flags =0 ; - pc->mem_cgroup =NULL ; pc->page =pfn_to_page (pfn); -INIT_LIST_HEAD (&pc->lru); +__init_mem_page_cgroup (pc); +__init_blkio_page_cgroup (pc); }static unsigned long total_usage;
@@ -74,7 +75,7 @@ void __init page_cgroup_init(void)int nid, fail; -if (mem_cgroup_disabled ()) +if (mem_cgroup_disabled () &&blkio_cgroup_disabled ())return ;for_each_online_node (nid) {
@@ -83,12 +84,12 @@ void __init page_cgroup_init(void) goto fail; }printk (KERN_INFO "allocated %ld bytes of page_cgroup\n" , total_usage); -printk (KERN_INFO "please try cgroup_disable=memory option if you" +printk (KERN_INFO "please try cgroup_disable=memory,blkio option if you" " don't want\n" );return ; fail:printk (KERN_CRIT "allocation of page_cgroup was failed.\n" ); -printk (KERN_CRIT "please try cgroup_disable=memory boot option\n" ); +printk (KERN_CRIT "please try cgroup_disable=memory,blkio boot options\n" );panic ("Out of memory" ); }
@@ -248,7 +249,7 @@ void __init page_cgroup_init(void)unsigned long pfn;int fail =0 ; -if (mem_cgroup_disabled ()) +if (mem_cgroup_disabled () &&blkio_cgroup_disabled ())return ;for (pfn =0 ; !fail && pfn < max_pfn; pfn +=PAGES_PER_SECTION ) {
@@ -263,8 +264,8 @@ void __init page_cgroup_init(void)hotplug_memory_notifier (page_cgroup_callback,0 ); }printk (KERN_INFO "allocated %ld bytes of page_cgroup\n" , total_usage); -printk (KERN_INFO "please try cgroup_disable=memory option if you don't" -" want\n" ); +printk (KERN_INFO "please try cgroup_disable=memory,blkio option" +" if you don't want\n" ); }void __meminit pgdat_page_cgroup_init (struct pglist_data *pgdat)
mm/swap_state.c +2 -0
@@ -18,6 +18,7 @@#include <linux/pagevec.h> #include <linux/migrate.h> #include <linux/page_cgroup.h> +#include <linux/biotrack.h> #include <asm/pgtable.h>
@@ -306,6 +307,7 @@ struct page *read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask,*/ __set_page_locked (new_page);SetPageSwapBacked (new_page); +blkio_cgroup_set_owner (new_page, current->mm); err =add_to_swap_cache (new_page, entry, gfp_mask &GFP_KERNEL );if (likely (!err)) {/*
Signature · 1 line
1.6.0.6
PATCH 15/20 io-controller: map async requests to appropriate cgroup
In reply to a message not in the archiveo So far we were assuming that a bio/rq belongs to the task who is submitting it. It did not hold good in case of async writes. This patch makes use of blkio_cgroup pataches to attribute the aysnc writes to right group instead of task submitting the bio.
o For sync requests, we continue to assume that io belongs to the task submitting it. Only in case of async requests, we make use of io tracking patches to track the owner cgroup.
o So far cfq always caches the async queue pointer. With async requests now not necessarily being tied to submitting task io context, caching the pointer will not help for async queues. This patch introduces a new config option CONFIG_TRACK_ASYNC_CONTEXT. If this option is not set, cfq retains old behavior where async queue pointer is cached in task context. If it is not set, async queue pointer is not cached and we take help of bio tracking patches to determine group bio belongs to and then map it to async queue of that group.
--- block/Kconfig.iosched | 16 +++++ block/as-iosched.c | 2 +- block/blk-core.c | 7 +- block/cfq-iosched.c | 152 ++++++++++++++++++++++++++++++++++++---------- block/deadline-iosched.c | 2 +- block/elevator-fq.c | 97 ++++++++++++++++++++++++----- block/elevator-fq.h | 23 ++++++- block/elevator.c | 15 +++-- include/linux/elevator.h | 21 ++++++- 9 files changed, 268 insertions(+), 67 deletions(-)
block/Kconfig.iosched +16 -0
@@ -124,6 +124,22 @@ config DEFAULT_IOSCHED default "cfq" if DEFAULT_CFQ default "noop" if DEFAULT_NOOP +config TRACK_ASYNC_CONTEXT + bool "Determine async request context from bio" + depends on GROUP_IOSCHED + select CGROUP_BLKIO + default n + ---help--- + Normally async request is attributed to the task submitting the + request. With group ioscheduling, for accurate accounting of + async writes, one needs to map the request to original task/cgroup + which originated the request and not the submitter of the request. + + Currently there are generic io tracking patches to provide facility + to map bio to original owner. If this option is set, for async + request, original owner of the bio is decided by using io tracking + patches otherwise we continue to attribute the request to the + submitting thread. endmenu endif
block/as-iosched.c +1 -1
@@ -1499,7 +1499,7 @@ as_merge(struct request_queue *q, struct request **req, struct bio *bio) {sector_t rb_key = bio->bi_sector +bio_sectors (bio);struct request *__rq; -struct as_queue *asq =elv_get_sched_queue_current (q); +struct as_queue *asq =elv_get_sched_queue_bio (q, bio);if (!asq)return ELEVATOR_NO_MERGE ;
block/blk-core.c +4 -3
@@ -643,7 +643,8 @@ static inline void blk_free_request(struct request_queue *q, struct request *rq) }static struct request * -blk_alloc_request (struct request_queue *q,int flags,int priv,gfp_t gfp_mask) +blk_alloc_request (struct request_queue *q,struct bio *bio,int flags,int priv, +gfp_t gfp_mask) {struct request *rq =mempool_alloc (q->rq.rq_pool, gfp_mask);
@@ -655,7 +656,7 @@ blk_alloc_request(struct request_queue *q, int flags, int priv, gfp_t gfp_mask) rq->cmd_flags = flags |REQ_ALLOCED ;if (priv) { -if (unlikely (elv_set_request (q, rq, gfp_mask))) { +if (unlikely (elv_set_request (q, rq, bio, gfp_mask))) {mempool_free (rq, q->rq.rq_pool);return NULL ; }
@@ -796,7 +797,7 @@ static struct request *get_request(struct request_queue *q, int rw_flags, rw_flags |=REQ_IO_STAT ;spin_unlock_irq (q->queue_lock); - rq =blk_alloc_request (q, rw_flags, priv, gfp_mask); + rq =blk_alloc_request (q, bio, rw_flags, priv, gfp_mask);if (unlikely (!rq)) {/* * Allocation failed presumably due to memory. Undo anything
block/cfq-iosched.c +120 -32
254 lines not on this page · raw message
block/deadline-iosched.c +1 -1
@@ -133,7 +133,7 @@ deadline_merge(struct request_queue *q, struct request **req, struct bio *bio)int ret;struct deadline_queue *dq; - dq =elv_get_sched_queue_current (q); + dq =elv_get_sched_queue_bio (q, bio);if (!dq)return ELEVATOR_NO_MERGE ;
block/elevator-fq.c +81 -16
185 lines not on this page · raw message
block/elevator-fq.h +19 -4
@@ -529,10 +529,12 @@ static inline int update_requeue(struct io_queue *ioq, int requeue) }extern int elv_fq_set_request_ioq (struct request_queue *q,struct request *rq, -gfp_t gfp_mask); +struct bio *bio,gfp_t gfp_mask);extern void elv_fq_unset_request_ioq (struct request_queue *q,struct request *rq);extern struct io_queue *elv_lookup_ioq_current (struct request_queue *q); +extern struct io_queue *elv_lookup_ioq_bio (struct request_queue *q, +struct bio *bio);/* Returns single ioq associated with the io group. */ static inline struct io_queue *io_group_ioq (struct io_group *iog)
@@ -590,7 +592,7 @@ static inline void io_group_set_ioq(struct io_group *iog, struct io_queue *ioq) }static inline int elv_fq_set_request_ioq (struct request_queue *q, -struct request *rq,gfp_t gfp_mask) +struct request *rq,struct bio *bio,gfp_t gfp_mask) {return 0 ; }
@@ -605,6 +607,12 @@ static inline struct io_queue *elv_lookup_ioq_current(struct request_queue *q) return NULL; } +static inline struct io_queue *elv_lookup_ioq_bio (struct request_queue *q, +struct bio *bio) +{ +return NULL ; +} +#endif /* GROUP_IOSCHED */ extern ssize_t elv_slice_idle_show (struct elevator_queue *q,char *name);
@@ -658,7 +666,8 @@ extern void *io_group_async_queue_prio(struct io_group *iog, int ioprio_class, int ioprio);extern void io_group_set_async_queue (struct io_group *iog,int ioprio_class,int ioprio,struct io_queue *ioq); -extern struct io_group *io_get_io_group (struct request_queue *q,int create); +extern struct io_group *io_get_io_group (struct request_queue *q, +struct bio *bio,int create);extern int elv_nr_busy_ioq (struct elevator_queue *e);extern struct io_queue *elv_alloc_ioq (struct request_queue *q,gfp_t gfp_mask);extern void elv_free_ioq (struct io_queue *ioq);
@@ -717,7 +726,7 @@ static inline int io_group_allow_merge(struct request *rq, struct bio *bio)return 1 ; }static inline int elv_fq_set_request_ioq (struct request_queue *q, -struct request *rq,gfp_t gfp_mask) +struct request *rq,struct bio *bio,gfp_t gfp_mask) {return 0 ; }
@@ -732,5 +741,11 @@ static inline struct io_queue *elv_lookup_ioq_current(struct request_queue *q)return NULL ; } +static inline struct io_queue *elv_lookup_ioq_bio (struct request_queue *q, +struct bio *bio) +{ +return NULL ; +} +#endif /* CONFIG_ELV_FAIR_QUEUING */ #endif /* _BFQ_SCHED_H */
block/elevator.c +8 -7
@@ -967,7 +967,8 @@ struct request *elv_former_request(struct request_queue *q, struct request *rq)return NULL ; } -int elv_set_request (struct request_queue *q,struct request *rq,gfp_t gfp_mask) +int elv_set_request (struct request_queue *q,struct request *rq, +struct bio *bio,gfp_t gfp_mask) {struct elevator_queue *e = q->elevator;
@@ -976,10 +977,10 @@ int elv_set_request(struct request_queue *q, struct request *rq, gfp_t gfp_mask)* ioq per io group */ if (elv_iosched_single_ioq (e)) -return elv_fq_set_request_ioq (q, rq, gfp_mask); +return elv_fq_set_request_ioq (q, rq, bio, gfp_mask);if (e->ops->elevator_set_req_fn) -return e->ops->elevator_set_req_fn (q, rq, gfp_mask); +return e->ops->elevator_set_req_fn (q, rq, bio, gfp_mask); rq->elevator_private =NULL ;return 0 ;
@@ -1368,19 +1369,19 @@ void *elv_select_sched_queue(struct request_queue *q, int force)EXPORT_SYMBOL (elv_select_sched_queue);/* -* Get the io scheduler queue pointer for current task. +* Get the io scheduler queue pointer for the group bio belongs to. * * If fair queuing is enabled, determine the io group of task and retrieve * the ioq pointer from that. This is used by only single queue ioschedulers * for retrieving the queue associated with the group to decide whether the * new bio can do a front merge or not. */ -void *elv_get_sched_queue_current (struct request_queue *q) +void *elv_get_sched_queue_bio (struct request_queue *q,struct bio *bio) {/* Fair queuing is not enabled. There is only one queue. */ if (!elv_iosched_fair_queuing_enabled (q->elevator))return q->elevator->sched_queue; -return ioq_sched_queue (elv_lookup_ioq_current (q)); +return ioq_sched_queue (elv_lookup_ioq_bio (q, bio)); } -EXPORT_SYMBOL (elv_get_sched_queue_current); +EXPORT_SYMBOL (elv_get_sched_queue_bio);
include/linux/elevator.h +18 -3
@@ -23,7 +23,7 @@ typedef struct request *(elevator_request_list_fn) (struct request_queue *, strutypedef void (elevator_completed_req_fn ) (struct request_queue *,struct request *);typedef int (elevator_may_queue_fn ) (struct request_queue *,int ); -typedef int (elevator_set_req_fn ) (struct request_queue *,struct request *,gfp_t ); +typedef int (elevator_set_req_fn ) (struct request_queue *,struct request *,struct bio *bio,gfp_t );typedef void (elevator_put_req_fn ) (struct request *);typedef void (elevator_activate_req_fn ) (struct request_queue *,struct request *);typedef void (elevator_deactivate_req_fn ) (struct request_queue *,struct request *);
@@ -150,7 +150,8 @@ extern void elv_unregister_queue(struct request_queue *q);extern int elv_may_queue (struct request_queue *,int );extern void elv_abort_queue (struct request_queue *);extern void elv_completed_request (struct request_queue *,struct request *); -extern int elv_set_request (struct request_queue *,struct request *,gfp_t ); +extern int elv_set_request (struct request_queue *,struct request *, +struct bio *bio,gfp_t );extern void elv_put_request (struct request_queue *,struct request *);extern void elv_drain_elevator (struct request_queue *);
@@ -279,6 +280,20 @@ static inline int elv_iosched_single_ioq(struct elevator_queue *e)#endif /* ELV_IOSCHED_FAIR_QUEUING */ extern void *elv_get_sched_queue (struct request_queue *q,struct request *rq);extern void *elv_select_sched_queue (struct request_queue *q,int force); -extern void *elv_get_sched_queue_current (struct request_queue *q); +extern void *elv_get_sched_queue_bio (struct request_queue *q,struct bio *bio); + +/* +* This is equivalent of rq_is_sync()/cfq_bio_sync() function where we +* determine whether an rq/bio is sync or not. There are cases like during +* merging and during * request allocation, where we don't have rq but bio +* and needs to find out * if this bio will be considered as sync or async by +* elevator/iosched. This function is useful in such cases. +*/ +static inline int elv_bio_sync (struct bio *bio) +{ +if ((bio_data_dir (bio) ==READ ) ||bio_sync (bio)) +return 1 ; +return 0 ; +}#endif /* CONFIG_BLOCK */ #endif
Signature · 1 line
1.6.0.6
PATCH 16/20 io-controller: Per cgroup request descriptor support
In reply to a message not in the archiveo Currently a request queue has got fixed number of request descriptors for sync and async requests. Once the request descriptors are consumed, new processes are put to sleep and they effectively become serialized. Because sync and async queues are separate, async requests don't impact sync ones but if one is looking for fairness between async requests, that is not achievable if request queue descriptors become bottleneck.
o Make request descriptor's per io group so that if there is lots of IO going on in one cgroup, it does not impact the IO of other group.
o This is just one relatively simple way of doing things. This patch will probably change after the feedback. Folks have raised concerns that in hierchical setup, child's request descriptors should be capped by parent's request descriptors. May be we need to have per cgroup per device files in cgroups where one can specify the upper limit of request descriptors and whenever a cgroup is created one needs to assign request descritor limit making sure total sum of child's request descriptor is not more than of parent.
I guess something like memory controller. Anyway, that would be the next step. For the time being, we have implemented something simpler as follows.
o This patch implements the per cgroup request descriptors. request pool per queue is still common but every group will have its own wait list and its own count of request descriptors allocated to that group for sync and async queues. So effectively request_list becomes per io group property and not a global request queue feature.
o Currently one can define q->nr_requests to limit request descriptors allocated for the queue. Now there is another tunable q->nr_group_requests which controls the requests descriptr limit per group. q->nr_requests supercedes q->nr_group_requests to make sure if there are lots of groups present, we don't end up allocating too many request descriptors on the queue.
o Issues: Currently notion of congestion is per queue. With per group request descriptor it is possible that queue is not congested but the group bio will go into is congested.
--- block/blk-core.c | 305 +++++++++++++++++++++++++++++++++++++---------- block/blk-settings.c | 1 + block/blk-sysfs.c | 58 +++++++-- block/elevator-fq.c | 14 +++ block/elevator-fq.h | 5 + block/elevator.c | 6 +- include/linux/blkdev.h | 87 +++++++++++++- 7 files changed, 394 insertions(+), 82 deletions(-)
block/blk-core.c +241 -66
@@ -480,20 +480,30 @@ void blk_cleanup_queue(struct request_queue *q) }EXPORT_SYMBOL (blk_cleanup_queue); -static int blk_init_free_list (struct request_queue *q) +void blk_init_request_list (struct request_list *rl) { -struct request_list *rl = &q->rq; rl->count[BLK_RW_SYNC ] = rl->count[BLK_RW_ASYNC ] =0 ; - rl->starved[BLK_RW_SYNC ] = rl->starved[BLK_RW_ASYNC ] =0 ; - rl->elvpriv =0 ;init_waitqueue_head (&rl->wait[BLK_RW_SYNC ]);init_waitqueue_head (&rl->wait[BLK_RW_ASYNC ]); +} - rl->rq_pool =mempool_create_node (BLKDEV_MIN_RQ , mempool_alloc_slab, - mempool_free_slab, request_cachep, q->node); +static int blk_init_free_list (struct request_queue *q) +{ +/* +* Initialize the queue request list in case there are non-hiearchical +* io schedulers not making use of fair queuing infrastructure. +* +* For ioschedulers making use of fair queuing infrastructure, request +* list is inside the associated group and when that group is +* instanciated, it takes care of initializing the request list also. +*/ +blk_init_request_list (&q->rq); + q->rq_data.rq_pool =mempool_create_node (BLKDEV_MIN_RQ , + mempool_alloc_slab, mempool_free_slab, + request_cachep, q->node); -if (!rl->rq_pool) +if (!q->rq_data.rq_pool)return -ENOMEM ;return 0 ;
@@ -590,6 +600,9 @@ blk_init_queue_node(request_fn_proc *rfn, spinlock_t *lock, int node_id)return NULL ; } +/* init starved waiter wait queue */ +init_waitqueue_head (&q->rq_data.starved_wait); +/* * if caller didn't supply a lock, they get per-queue locking with * our embedded lock
@@ -639,14 +652,14 @@ static inline void blk_free_request(struct request_queue *q, struct request *rq) {if (rq->cmd_flags &REQ_ELVPRIV )elv_put_request (q, rq); -mempool_free (rq, q->rq.rq_pool); +mempool_free (rq, q->rq_data.rq_pool); }static struct request *blk_alloc_request (struct request_queue *q,struct bio *bio,int flags,int priv,gfp_t gfp_mask) { -struct request *rq =mempool_alloc (q->rq.rq_pool, gfp_mask); +struct request *rq =mempool_alloc (q->rq_data.rq_pool, gfp_mask);if (!rq)return NULL ;
@@ -657,7 +670,7 @@ blk_alloc_request(struct request_queue *q, struct bio *bio, int flags, int priv,if (priv) {if (unlikely (elv_set_request (q, rq, bio, gfp_mask))) { -mempool_free (rq, q->rq.rq_pool); +mempool_free (rq, q->rq_data.rq_pool);return NULL ; } rq->cmd_flags |=REQ_ELVPRIV ;
@@ -700,18 +713,18 @@ static void ioc_set_batching(struct request_queue *q, struct io_context *ioc) ioc->last_waited = jiffies; } -static void __freed_request (struct request_queue *q,int sync) +static void __freed_request (struct request_queue *q,int sync, +struct request_list *rl) { -struct request_list *rl = &q->rq; - -if (rl->count[sync] <queue_congestion_off_threshold (q)) +if (q->rq_data.count[sync] <queue_congestion_off_threshold (q))blk_clear_queue_congested (q, sync); -if (rl->count[sync] +1 <= q->nr_requests) { +if (q->rq_data.count[sync] +1 <= q->nr_requests) +blk_clear_queue_full (q, sync); + +if (rl->count[sync] +1 <= q->nr_group_requests) {if (waitqueue_active (&rl->wait[sync]))wake_up (&rl->wait[sync]); - -blk_clear_queue_full (q, sync); } }
@@ -719,63 +732,133 @@ static void __freed_request(struct request_queue *q, int sync)* A request has just been released. Account for it, update the full and * congestion status, wake up any waiters. Called under q->queue_lock. */ -static void freed_request (struct request_queue *q,int sync,int priv) -{ -struct request_list *rl = &q->rq; +static void freed_request (struct request_queue *q,int sync,int priv, +struct request_list *rl) +{ +/* There is a window during request allocation where request is +* mapped to one group but by the time a queue for the group is +* allocated, it is possible that original cgroup/io group has been +* deleted and now io queue is allocated in a different group (root) +* altogether. +* +* One solution to the problem is that rq should take io group +* reference. But it looks too much to do that to solve this issue. +* The only side affect to the hard to hit issue seems to be that +* we will try to decrement the rl->count for a request list which +* did not allocate that request. Chcek for rl->count going less than +* zero and do not decrement it if that's the case. +*/ + +if (priv && rl->count[sync] >0 ) + rl->count[sync]--; + +BUG_ON (!q->rq_data.count[sync]); + q->rq_data.count[sync]--; - rl->count[sync]--;if (priv) - rl->elvpriv--; + q->rq_data.elvpriv--; -__freed_request (q, sync); +__freed_request (q, sync, rl);if (unlikely (rl->starved[sync ^1 ])) -__freed_request (q, sync ^1 ); +__freed_request (q, sync ^1 , rl); + +/* Wake up the starved process on global list, if any */ +if (unlikely (q->rq_data.starved)) { +if (waitqueue_active (&q->rq_data.starved_wait)) +wake_up (&q->rq_data.starved_wait); + q->rq_data.starved--; + } +} + +/* +* Returns whether one can sleep on this request list or not. There are +* cases (elevator switch) where request list might not have allocated +* any request descriptor but we deny request allocation due to gloabl +* limits. In that case one should sleep on global list as on this request +* list no wakeup will take place. +* +* Also sets the request list starved flag if there are no requests pending +* in the direction of rq. +* +* Return 1 --> sleep on request list, 0 --> sleep on global list +*/ +static int can_sleep_on_request_list (struct request_list *rl,int is_sync) +{ +if (unlikely (rl->count[is_sync] ==0 )) { +/* +* If there is a request pending in other direction +* in same io group, then set the starved flag of +* the group request list. Otherwise, we need to +* make this process sleep in global starved list +* to make sure it will not sleep indefinitely. +*/ +if (rl->count[is_sync ^1 ] !=0 ) { + rl->starved[is_sync] =1 ; +return 1 ; + }else +return 0 ; + } + +return 1 ; }/* * Get a free request, queue_lock must be held. -* Returns NULL on failure, with queue_lock held. +* Returns NULL on failure, with queue_lock held. Also sets the "reason" field +* in case of failure. This reason field helps caller decide to whether sleep +* on per group list or global per queue list. +* reason = 0 sleep on per group list +* reason = 1 sleep on global list +* * Returns !NULL on success, with queue_lock *not held*. */ static struct request *get_request (struct request_queue *q,int rw_flags, -struct bio *bio,gfp_t gfp_mask) +struct bio *bio,gfp_t gfp_mask, +struct request_list *rl,int *reason) {struct request *rq =NULL ; -struct request_list *rl = &q->rq;struct io_context *ioc =NULL ;const bool is_sync =rw_is_sync (rw_flags) !=0 ;int may_queue, priv; +int sleep_on_global =0 ; may_queue =elv_may_queue (q, rw_flags);if (may_queue ==ELV_MQUEUE_NO ) goto rq_starved; -if (rl->count[is_sync]+1 >=queue_congestion_on_threshold (q)) { -if (rl->count[is_sync]+1 >= q->nr_requests) { - ioc =current_io_context (GFP_ATOMIC , q->node); -/* -* The queue will fill after this allocation, so set -* it as full, and mark this process as "batching". -* This process will be allowed to complete a batch of -* requests, others will be blocked. -*/ -if (!blk_queue_full (q, is_sync)) { -ioc_set_batching (q, ioc); -blk_set_queue_full (q, is_sync); - }else { -if (may_queue !=ELV_MQUEUE_MUST - && !ioc_batching (q, ioc)) { -/* -* The queue is full and the allocating -* process is not a "batcher", and not -* exempted by the IO scheduler -*/ - goto out; - } +if (q->rq_data.count[is_sync]+1 >=queue_congestion_on_threshold (q)) +blk_set_queue_congested (q, is_sync); + +/* +* Looks like there is no user of queue full now. +* Keeping it for time being. +*/ +if (q->rq_data.count[is_sync]+1 >= q->nr_requests) +blk_set_queue_full (q, is_sync); + +if (rl->count[is_sync]+1 >= q->nr_group_requests) { + ioc =current_io_context (GFP_ATOMIC , q->node); +/* +* The queue request descriptor group will fill after this +* allocation, so set +* it as full, and mark this process as "batching". +* This process will be allowed to complete a batch of +* requests, others will be blocked. +*/ +if (rl->count[is_sync] <= q->nr_group_requests) +ioc_set_batching (q, ioc); +else { +if (may_queue !=ELV_MQUEUE_MUST + && !ioc_batching (q, ioc)) { +/* +* The queue is full and the allocating +* process is not a "batcher", and not +* exempted by the IO scheduler +*/ + goto out; } } -blk_set_queue_congested (q, is_sync); }/*
@@ -783,21 +866,60 @@ static struct request *get_request(struct request_queue *q, int rw_flags,* limit of requests, otherwise we could have thousands of requests * allocated with any setting of ->nr_requests */ -if (rl->count[is_sync] >= (3 * q->nr_requests /2 )) + +if (q->rq_data.count[is_sync] >= (3 * q->nr_requests /2 )) { +/* +* Queue is too full for allocation. On which request queue +* the task should sleep? Generally it should sleep on its +* request list but if elevator switch is happening, in that +* window, request descriptors are allocated from global +* pool and are not accounted against any particular request +* list as group is going away. +* +* So it might happen that request list does not have any +* requests allocated at all and if process sleeps on per +* group request list, it will not be woken up. In such case, +* make it sleep on global starved list. +*/ +if (test_bit (QUEUE_FLAG_ELVSWITCH , &q->queue_flags) + || !can_sleep_on_request_list (rl, is_sync)) + sleep_on_global =1 ; + goto out; + } + +/* +* Allocation of request is allowed from queue perspective. Now check +* from per group request list +*/ + +if (rl->count[is_sync] >= (3 * q->nr_group_requests /2 )) goto out; - rl->count[is_sync]++; rl->starved[is_sync] =0 ; + q->rq_data.count[is_sync]++; + priv = !test_bit (QUEUE_FLAG_ELVSWITCH , &q->queue_flags); -if (priv) - rl->elvpriv++; +if (priv) { + q->rq_data.elvpriv++; +/* +* Account the request to request list only if request is +* going to elevator. During elevator switch, there will +* be small window where group is going away and new group +* will not be allocated till elevator switch is complete. +* So till then instead of slowing down the application, +* we will continue to allocate request from total common +* pool instead of per group limit +*/ + rl->count[is_sync]++; + }if (blk_queue_io_stat (q)) rw_flags |=REQ_IO_STAT ;spin_unlock_irq (q->queue_lock); rq =blk_alloc_request (q, bio, rw_flags, priv, gfp_mask); +if (unlikely (!rq)) {/* * Allocation failed presumably due to memory. Undo anything
@@ -807,7 +929,7 @@ static struct request *get_request(struct request_queue *q, int rw_flags,* wait queue, but this is pretty rare. */ spin_lock_irq (q->queue_lock); -freed_request (q, is_sync, priv); +freed_request (q, is_sync, priv, rl);/* * in the very unlikely event that allocation failed and no
@@ -817,9 +939,8 @@ static struct request *get_request(struct request_queue *q, int rw_flags,* rq mempool into READ and WRITE */ rq_starved: -if (unlikely (rl->count[is_sync] ==0 )) - rl->starved[is_sync] =1 ; - +if (!can_sleep_on_request_list (rl, is_sync)) + sleep_on_global =1 ; goto out; }
@@ -834,6 +955,8 @@ rq_starved:trace_block_getrq (q, bio, rw_flags &1 ); out: +if (reason && sleep_on_global) + *reason =1 ;return rq; }
@@ -847,16 +970,44 @@ static struct request *get_request_wait(struct request_queue *q, int rw_flags, struct bio *bio) {const bool is_sync =rw_is_sync (rw_flags) !=0 ; +int sleep_on_global =0 ;struct request *rq; +struct request_list *rl =blk_get_request_list (q, bio); +struct io_group *iog =NULL ; - rq =get_request (q, rw_flags, bio,GFP_NOIO ); + rq =get_request (q, rw_flags, bio,GFP_NOIO , rl, &sleep_on_global);while (!rq) {DEFINE_WAIT (wait);struct io_context *ioc; -struct request_list *rl = &q->rq; -prepare_to_wait_exclusive (&rl->wait[is_sync], &wait, -TASK_UNINTERRUPTIBLE ); +if (sleep_on_global) { +/* +* Task failed allocation and needs to wait and +* try again. There are no requests pending from +* the io group hence need to sleep on global +* wait queue. Most likely the allocation failed +* because of memory issues. +*/ + + q->rq_data.starved++; +prepare_to_wait_exclusive (&q->rq_data.starved_wait, + &wait,TASK_UNINTERRUPTIBLE ); + }else { +/* +* We are about to sleep on a request list and we +* drop queue lock. After waking up, we will do +* finish_wait() on request list and in the mean +* time group might be gone. Take a reference to +* the group now. +*/ +prepare_to_wait_exclusive (&rl->wait[is_sync], &wait, +TASK_UNINTERRUPTIBLE ); +#ifdef CONFIG_GROUP_IOSCHED + iog =rl_iog (rl); +if (iog) +elv_get_iog (iog); +#endif + }trace_block_sleeprq (q, bio, rw_flags &1 );
@@ -874,9 +1025,30 @@ static struct request *get_request_wait(struct request_queue *q, int rw_flags,ioc_set_batching (q, ioc);spin_lock_irq (q->queue_lock); -finish_wait (&rl->wait[is_sync], &wait); - rq =get_request (q, rw_flags, bio,GFP_NOIO ); +if (sleep_on_global) { +finish_wait (&q->rq_data.starved_wait, &wait); + sleep_on_global =0 ; + }else { +finish_wait (&rl->wait[is_sync], &wait); +#ifdef CONFIG_GROUP_IOSCHED +/* +* We had taken a reference to the rl/iog. +* Put that now +*/ + iog =rl_iog (rl); +if (iog) +elv_put_iog (iog); +#endif + } + +/* +* After the sleep check the rl again in case cgrop bio +* belonged to is gone and it is mapped to root group now +*/ + rl =blk_get_request_list (q, bio); + rq =get_request (q, rw_flags, bio,GFP_NOIO , rl, + &sleep_on_global); };return rq;
@@ -885,14 +1057,16 @@ static struct request *get_request_wait(struct request_queue *q, int rw_flags,struct request *blk_get_request (struct request_queue *q,int rw,gfp_t gfp_mask) {struct request *rq; +struct request_list *rl;BUG_ON (rw !=READ && rw !=WRITE );spin_lock_irq (q->queue_lock); + rl =blk_get_request_list (q,NULL );if (gfp_mask & __GFP_WAIT) { rq =get_request_wait (q, rw,NULL ); }else { - rq =get_request (q, rw,NULL , gfp_mask); + rq =get_request (q, rw,NULL , gfp_mask, rl,NULL );if (!rq)spin_unlock_irq (q->queue_lock); }
@@ -1075,12 +1249,13 @@ void __blk_put_request(struct request_queue *q, struct request *req)if (req->cmd_flags &REQ_ALLOCED ) {int is_sync =rq_is_sync (req) !=0 ;int priv = req->cmd_flags &REQ_ELVPRIV ; +struct request_list *rl =rq_rl (q, req);BUG_ON (!list_empty (&req->queuelist));BUG_ON (!hlist_unhashed (&req->hash));blk_free_request (q, req); -freed_request (q, is_sync, priv); +freed_request (q, is_sync, priv, rl); } }EXPORT_SYMBOL_GPL (__blk_put_request);
block/blk-settings.c +1 -0
@@ -123,6 +123,7 @@ void blk_queue_make_request(struct request_queue *q, make_request_fn *mfn)* set defaults */ q->nr_requests =BLKDEV_MAX_RQ ; + q->nr_group_requests =BLKDEV_MAX_GROUP_RQ ;blk_queue_max_phys_segments (q,MAX_PHYS_SEGMENTS );blk_queue_max_hw_segments (q,MAX_HW_SEGMENTS );blk_queue_segment_boundary (q,BLK_SEG_BOUNDARY_MASK );
block/blk-sysfs.c +46 -12
112 lines not on this page · raw message
block/elevator-fq.c +14 -0
@@ -1082,6 +1082,16 @@ struct io_cgroup *cgroup_to_io_cgroup(struct cgroup *cgroup) struct io_cgroup, css); } +struct request_list *io_group_get_request_list (struct request_queue *q, +struct bio *bio) +{ +struct io_group *iog; + + iog =io_get_io_group (q, bio,1 ); +BUG_ON (!iog); +return &iog->rl; +} +/* * Search the bfq_group for bfqd into the hash table (by now only a list) * of bgrp. Must be called under rcu_read_lock().
@@ -1297,6 +1307,8 @@ struct io_group *io_group_chain_alloc(struct request_queue *q, void *key,*/ elv_get_iog (iog); +blk_init_request_list (&iog->rl); +if (leaf ==NULL ) { leaf = iog; prev = leaf;
@@ -1557,6 +1569,8 @@ struct io_group *io_alloc_root_group(struct request_queue *q,for (i =0 ; i <IO_IOPRIO_CLASSES ; i++) iog->sched_data.service_tree[i] =IO_SERVICE_TREE_INIT ; +blk_init_request_list (&iog->rl); + iocg = &io_root_cgroup;spin_lock_irq (&iocg->lock);rcu_assign_pointer (iog->key, key);
block/elevator-fq.h +5 -0
@@ -257,6 +257,9 @@ struct io_group {/* Single ioq per group, used for noop, deadline, anticipatory */ struct io_queue *ioq; + +/* request list associated with the group */ +struct request_list rl; };/**
@@ -535,6 +538,8 @@ extern void elv_fq_unset_request_ioq(struct request_queue *q,extern struct io_queue *elv_lookup_ioq_current (struct request_queue *q);extern struct io_queue *elv_lookup_ioq_bio (struct request_queue *q,struct bio *bio); +extern struct request_list *io_group_get_request_list (struct request_queue *q, +struct bio *bio);/* Returns single ioq associated with the io group. */ static inline struct io_queue *io_group_ioq (struct io_group *iog)
block/elevator.c +3 -3
@@ -668,7 +668,7 @@ void elv_quiesce_start(struct request_queue *q)* make sure we don't have any requests in flight */ elv_drain_elevator (q); -while (q->rq.elvpriv) { +while (q->rq_data.elvpriv) {blk_start_queueing (q);spin_unlock_irq (q->queue_lock);msleep (10 );
@@ -768,8 +768,8 @@ void elv_insert(struct request_queue *q, struct request *rq, int where) }if (unplug_it &&blk_queue_plugged (q)) { -int nrq = q->rq.count[BLK_RW_SYNC ] + q->rq.count[BLK_RW_ASYNC ] - - q->in_flight; +int nrq = q->rq_data.count[BLK_RW_SYNC ] + + q->rq_data.count[BLK_RW_ASYNC ] - q->in_flight;if (nrq >= q->unplug_thresh)__generic_unplug_device (q);
include/linux/blkdev.h +85 -2
124 lines not on this page · raw message
Signature · 1 line
1.6.0.6
PATCH 17/20 io-controller: Per io group bdi congestion interface
In reply to a message not in the archiveo So far there used to be only one pair or queue of request descriptors (one for sync and one for async) per device and number of requests allocated used to decide whether associated bdi is congested or not.
Now with per io group request descriptor infrastructure, there is a pair of request descriptor queue per io group per device. So it might happen that overall request queue is not congested but a particular io group bio belongs to is congested.
Or, it could be otherwise that group is not congested but overall queue is congested. This can happen if user has not properly set the request descriptors limits for queue and groups.
(q->nr_requests < nr_groups * q->nr_group_requests)
Hence there is a need for new interface which can query deivce congestion status per group. This group is determined by the "struct page" IO will be done for. If page is null, then group is determined from the current task context.
o This patch introduces new set of function bdi_*_congested_group(), which take "struct page" as addition argument. These functions will call the block layer and in trun elevator to find out if the io group the page will go into is congested or not.
o Currently I have introduced the core functions and migrated most of the users. But there might be still some left. This is an ongoing TODO item.
o There are some io_get_io_group() related changes which should be pushed into higher patches. Still testing this patch. Will push these changes up in next posting.
--- block/blk-core.c | 21 +++++ block/cfq-iosched.c | 6 +- block/elevator-fq.c | 179 ++++++++++++++++++++++++++++++++++--------- block/elevator-fq.h | 10 ++- drivers/md/dm-table.c | 11 ++- drivers/md/dm.c | 5 +- drivers/md/dm.h | 3 +- drivers/md/linear.c | 7 +- drivers/md/multipath.c | 7 +- drivers/md/raid0.c | 6 +- drivers/md/raid1.c | 9 ++- drivers/md/raid10.c | 6 +- drivers/md/raid5.c | 2 +- fs/afs/write.c | 8 ++- fs/btrfs/disk-io.c | 6 +- fs/btrfs/extent_io.c | 12 +++ fs/btrfs/volumes.c | 8 ++- fs/cifs/file.c | 11 +++ fs/ext2/ialloc.c | 2 +- fs/gfs2/ops_address.c | 12 +++ fs/nilfs2/segbuf.c | 3 +- fs/xfs/linux-2.6/xfs_aops.c | 2 +- fs/xfs/linux-2.6/xfs_buf.c | 2 +- include/linux/backing-dev.h | 61 ++++++++++++++- include/linux/biotrack.h | 6 ++ include/linux/blkdev.h | 5 + mm/backing-dev.c | 62 +++++++++++++++ mm/biotrack.c | 21 +++++ mm/page-writeback.c | 11 +++ mm/readahead.c | 4 +- 30 files changed, 435 insertions(+), 73 deletions(-)
block/blk-core.c +21 -0
@@ -99,6 +99,27 @@ void blk_queue_congestion_threshold(struct request_queue *q) q->nr_congestion_off = nr; } +#ifdef CONFIG_GROUP_IOSCHED +int blk_queue_io_group_congested (struct backing_dev_info *bdi,int bdi_bits, +struct page *page) +{ +int ret =0 ; +struct request_queue *q = bdi->unplug_io_data; + +if (!q && !q->elevator) +return bdi_congested (bdi, bdi_bits); + +/* Do we need to hold queue lock? */ +if (bdi_bits & (1 << BDI_sync_congested)) + ret |=elv_io_group_congested (q, page,1 ); + +if (bdi_bits & (1 << BDI_async_congested)) + ret |=elv_io_group_congested (q, page,0 ); + +return ret; +} +#endif +/** * blk_get_backing_dev_info - get the address of a queue's backing_dev_info * @bdev: device
block/cfq-iosched.c +3 -3
@@ -195,7 +195,7 @@ static struct cfq_queue *cic_bio_to_cfqq(struct cfq_data *cfqd,* async bio tracking is enabled and we are not caching * async queue pointer in cic. */ - iog =io_get_io_group (cfqd->queue, bio,0 ); + iog =io_get_io_group_bio (cfqd->queue, bio,0 );if (!iog) {/* * May be this is first rq/bio and io group has not
@@ -1334,7 +1334,7 @@ cfq_find_alloc_queue(struct cfq_data *cfqd, struct bio *bio, int is_sync,struct io_queue *ioq =NULL , *new_ioq =NULL ;struct io_group *iog =NULL ; retry: - iog =io_get_io_group (q, bio,1 ); + iog =io_get_io_group_bio (q, bio,1 ); cic =cfq_cic_lookup (cfqd, ioc);/* cic always exists here */
@@ -1452,7 +1452,7 @@ cfq_get_queue(struct cfq_data *cfqd, struct bio *bio, int is_sync,const int ioprio_class =task_ioprio_class (ioc);struct cfq_queue *async_cfqq =NULL ;struct cfq_queue *cfqq =NULL ; -struct io_group *iog =io_get_io_group (cfqd->queue, bio,1 ); +struct io_group *iog =io_get_io_group_bio (cfqd->queue, bio,1 );if (!is_sync) { async_cfqq = io_group_async_queue_prio(iog, ioprio_class,
block/elevator-fq.c +142 -37
319 lines not on this page · raw message
block/elevator-fq.h +9 -1
@@ -258,8 +258,13 @@ struct io_group {/* Single ioq per group, used for noop, deadline, anticipatory */ struct io_queue *ioq; +/* io group congestion on and off threshold for request descriptors */ +unsigned int nr_congestion_on; +unsigned int nr_congestion_off; +/* request list associated with the group */ struct request_list rl; +struct rcu_head rcu_head; };/**
@@ -540,7 +545,8 @@ extern struct io_queue *elv_lookup_ioq_bio(struct request_queue *q, struct bio *bio);extern struct request_list *io_group_get_request_list (struct request_queue *q,struct bio *bio); - +extern int elv_io_group_congested (struct request_queue *q,struct page *page, +int sync);/* Returns single ioq associated with the io group. */ static inline struct io_queue *io_group_ioq (struct io_group *iog) {
@@ -672,6 +678,8 @@ extern void *io_group_async_queue_prio(struct io_group *iog, int ioprio_class,extern void io_group_set_async_queue (struct io_group *iog,int ioprio_class,int ioprio,struct io_queue *ioq);extern struct io_group *io_get_io_group (struct request_queue *q, +struct page *page,int create); +extern struct io_group *io_get_io_group_bio (struct request_queue *q,struct bio *bio,int create);extern int elv_nr_busy_ioq (struct elevator_queue *e);extern struct io_queue *elv_alloc_ioq (struct request_queue *q,gfp_t gfp_mask);
drivers/md/dm-table.c +7 -4
@@ -1000,7 +1000,8 @@ int dm_table_resume_targets(struct dm_table *t)return 0 ; } -int dm_table_any_congested (struct dm_table *t,int bdi_bits) +int dm_table_any_congested (struct dm_table *t,int bdi_bits,struct page *page, +int group) {struct dm_dev_internal *dd;struct list_head *devices =dm_table_get_devices (t);
@@ -1010,9 +1011,11 @@ int dm_table_any_congested(struct dm_table *t, int bdi_bits)struct request_queue *q =bdev_get_queue (dd->dm_dev.bdev);char b[BDEVNAME_SIZE ]; -if (likely (q)) - r |=bdi_congested (&q->backing_dev_info, bdi_bits); -else +if (likely (q)) { +struct backing_dev_info *bdi = &q->backing_dev_info; + r |= group ?bdi_congested_group (bdi, bdi_bits, page) + :bdi_congested (bdi, bdi_bits); + }else DMWARN_LIMIT ("%s: any_congested: nonexistent device %s" ,dm_device_name (t->md),bdevname (dd->dm_dev.bdev, b));
drivers/md/dm.c +3 -2
@@ -994,7 +994,8 @@ static void dm_unplug_all(struct request_queue *q) } } -static int dm_any_congested (void *congested_data,int bdi_bits) +static int dm_any_congested (void *congested_data,int bdi_bits, +struct page *page,int group) {int r = bdi_bits;struct mapped_device *md = congested_data;
@@ -1003,7 +1004,7 @@ static int dm_any_congested(void *congested_data, int bdi_bits)if (!test_bit (DMF_BLOCK_IO_FOR_SUSPEND , &md->flags)) { map =dm_get_table (md);if (map) { - r =dm_table_any_congested (map, bdi_bits); + r =dm_table_any_congested (map, bdi_bits, page, group);dm_table_put (map); } }
drivers/md/dm.h +2 -1
@@ -46,7 +46,8 @@ struct list_head *dm_table_get_devices(struct dm_table *t);void dm_table_presuspend_targets (struct dm_table *t);void dm_table_postsuspend_targets (struct dm_table *t);int dm_table_resume_targets (struct dm_table *t); -int dm_table_any_congested (struct dm_table *t,int bdi_bits); +int dm_table_any_congested (struct dm_table *t,int bdi_bits,struct page *page, +int group);/* * To check the return value from dm_table_find_target().
drivers/md/linear.c +5 -2
@@ -88,7 +88,7 @@ static void linear_unplug(struct request_queue *q) } } -static int linear_congested (void *data,int bits) +static int linear_congested (void *data,int bits,struct page *page,int group) {mddev_t *mddev = data;linear_conf_t *conf =mddev_to_conf (mddev);
@@ -96,7 +96,10 @@ static int linear_congested(void *data, int bits)for (i =0 ; i < mddev->raid_disks && !ret ; i++) {struct request_queue *q =bdev_get_queue (conf->disks[i].rdev->bdev); - ret |=bdi_congested (&q->backing_dev_info, bits); +struct backing_dev_info *bdi = &q->backing_dev_info; + + ret |= group ?bdi_congested_group (bdi, bits, page) : +bdi_congested (bdi, bits); }return ret; }
drivers/md/multipath.c +5 -2
@@ -192,7 +192,8 @@ static void multipath_status (struct seq_file *seq, mddev_t *mddev)seq_printf (seq,"]" ); } -static int multipath_congested (void *data,int bits) +static int multipath_congested (void *data,int bits,struct page *page, +int group) {mddev_t *mddev = data;multipath_conf_t *conf =mddev_to_conf (mddev);
@@ -203,8 +204,10 @@ static int multipath_congested(void *data, int bits)mdk_rdev_t *rdev =rcu_dereference (conf->multipaths[i].rdev);if (rdev && !test_bit (Faulty, &rdev->flags)) {struct request_queue *q =bdev_get_queue (rdev->bdev); +struct backing_dev_info *bdi = &q->backing_dev_info; - ret |=bdi_congested (&q->backing_dev_info, bits); + ret |= group ?bdi_congested_group (bdi, bits, page) + :bdi_congested (bdi, bits);/* Just like multipath_map, we just check the * first available device */
drivers/md/raid0.c +4 -2
@@ -37,7 +37,7 @@ static void raid0_unplug(struct request_queue *q) } } -static int raid0_congested (void *data,int bits) +static int raid0_congested (void *data,int bits,struct page *page,int group) {mddev_t *mddev = data;raid0_conf_t *conf =mddev_to_conf (mddev);
@@ -46,8 +46,10 @@ static int raid0_congested(void *data, int bits)for (i =0 ; i < mddev->raid_disks && !ret ; i++) {struct request_queue *q =bdev_get_queue (devlist[i]->bdev); +struct backing_dev_info *bdi = &q->backing_dev_info; - ret |=bdi_congested (&q->backing_dev_info, bits); + ret |= group ?bdi_congested_group (bdi, bits, page) + :bdi_congested (bdi, bits); }return ret; }
drivers/md/raid1.c +6 -3
@@ -570,7 +570,7 @@ static void raid1_unplug(struct request_queue *q)md_wakeup_thread (mddev->thread); } -static int raid1_congested (void *data,int bits) +static int raid1_congested (void *data,int bits,struct page *page,int group) {mddev_t *mddev = data;conf_t *conf =mddev_to_conf (mddev);
@@ -581,14 +581,17 @@ static int raid1_congested(void *data, int bits)mdk_rdev_t *rdev =rcu_dereference (conf->mirrors[i].rdev);if (rdev && !test_bit (Faulty, &rdev->flags)) {struct request_queue *q =bdev_get_queue (rdev->bdev); +struct backing_dev_info *bdi = &q->backing_dev_info;/* Note the '|| 1' - when read_balance prefers * non-congested targets, it can be removed */ if ((bits & (1 <<BDI_async_congested)) ||1 ) - ret |=bdi_congested (&q->backing_dev_info, bits); + ret |= group ?bdi_congested_group (bdi, bits, + page) :bdi_congested (bdi, bits);else - ret &=bdi_congested (&q->backing_dev_info, bits); + ret &= group ?bdi_congested_group (bdi, bits, + page) :bdi_congested (bdi, bits); } }rcu_read_unlock ();
drivers/md/raid10.c +4 -2
@@ -625,7 +625,7 @@ static void raid10_unplug(struct request_queue *q)md_wakeup_thread (mddev->thread); } -static int raid10_congested (void *data,int bits) +static int raid10_congested (void *data,int bits,struct page *page,int group) {mddev_t *mddev = data;conf_t *conf =mddev_to_conf (mddev);
@@ -636,8 +636,10 @@ static int raid10_congested(void *data, int bits)mdk_rdev_t *rdev =rcu_dereference (conf->mirrors[i].rdev);if (rdev && !test_bit (Faulty, &rdev->flags)) {struct request_queue *q =bdev_get_queue (rdev->bdev); +struct backing_dev_info *bdi = &q->backing_dev_info; - ret |=bdi_congested (&q->backing_dev_info, bits); + ret |= group ?bdi_congested_group (bdi, bits, page) + :bdi_congested (bdi, bits); } }rcu_read_unlock ();
drivers/md/raid5.c +1 -1
@@ -3324,7 +3324,7 @@ static void raid5_unplug_device(struct request_queue *q)unplug_slaves (mddev); } -static int raid5_congested (void *data,int bits) +static int raid5_congested (void *data,int bits,struct page *page,int group) {mddev_t *mddev = data;raid5_conf_t *conf =mddev_to_conf (mddev);
fs/afs/write.c +7 -1
@@ -455,7 +455,7 @@ int afs_writepage(struct page *page, struct writeback_control *wbc) } wbc->nr_to_write -= ret; -if (wbc->nonblocking &&bdi_write_congested (bdi)) +if (wbc->nonblocking &&bdi_or_group_write_congested (bdi, page)) wbc->encountered_congestion =1 ;_leave (" = 0" );
@@ -491,6 +491,12 @@ static int afs_writepages_region(struct address_space *mapping,return 0 ; } +if (wbc->nonblocking &&bdi_write_congested_group (bdi, page)) { + wbc->encountered_congestion =1 ; +page_cache_release (page); +break ; + } +/* at this point we hold neither mapping->tree_lock nor lock on * the page itself: the page may be truncated or invalidated * (changing page->mapping to NULL), or even swizzled back from
fs/btrfs/disk-io.c +4 -2
@@ -1250,7 +1250,8 @@ struct btrfs_root *btrfs_read_fs_root(struct btrfs_fs_info *fs_info,return root; } -static int btrfs_congested_fn (void *congested_data,int bdi_bits) +static int btrfs_congested_fn (void *congested_data,int bdi_bits, +struct page *page,int group) {struct btrfs_fs_info *info = (struct btrfs_fs_info *)congested_data;int ret =0 ;
@@ -1261,7 +1262,8 @@ static int btrfs_congested_fn(void *congested_data, int bdi_bits)if (!device->bdev)continue ; bdi =blk_get_backing_dev_info (device->bdev); -if (bdi &&bdi_congested (bdi, bdi_bits)) { +if (bdi && (group ?bdi_congested_group (bdi, bdi_bits, page) : +bdi_congested (bdi, bdi_bits))) { ret =1 ;break ; }
fs/btrfs/extent_io.c +12 -0
@@ -2358,6 +2358,18 @@ retry:unsigned i; scanned =1 ; + +/* +* If the io group page will go into is congested, bail out. +*/ +if (wbc->nonblocking + &&bdi_write_congested_group (bdi, pvec.pages[0 ])) { + wbc->encountered_congestion =1 ; + done =1 ; +pagevec_release (&pvec); +break ; + } +for (i =0 ; i < nr_pages; i++) {struct page *page = pvec.pages[i];
fs/btrfs/volumes.c +6 -2
@@ -163,6 +163,7 @@ static noinline int run_scheduled_bios(struct btrfs_device *device)unsigned long num_sync_run;unsigned long limit;unsigned long last_waited =0 ; +struct page *page; bdi =blk_get_backing_dev_info (device->bdev); fs_info = device->dev_root->fs_info;
@@ -265,8 +266,11 @@ loop_lock:* is now congested. Back off and let other work structs * run instead */ -if (pending &&bdi_write_congested (bdi) && num_run >16 && - fs_info->fs_devices->open_devices >1 ) { +if (pending) + page =bio_iovec_idx (pending,0 )->bv_page; + +if (pending &&bdi_or_group_write_congested (bdi, page) && + num_run >16 && fs_info->fs_devices->open_devices >1 ) {struct io_context *ioc; ioc = current->io_context;
fs/cifs/file.c +11 -0
@@ -1466,6 +1466,17 @@ retry: n_iov =0 ; bytes_to_write =0 ; +/* +* If the io group page will go into is congested, bail out. +*/ +if (wbc->nonblocking && +bdi_write_congested_group (bdi, pvec.pages[0 ])) { + wbc->encountered_congestion =1 ; + done =1 ; +pagevec_release (&pvec); +break ; + } +for (i =0 ; i < nr_pages; i++) { page = pvec.pages[i];/*
fs/ext2/ialloc.c +1 -1
@@ -179,7 +179,7 @@ static void ext2_preread_inode(struct inode *inode)struct backing_dev_info *bdi; bdi = inode->i_mapping->backing_dev_info; -if (bdi_read_congested (bdi)) +if (bdi_or_group_read_congested (bdi,NULL ))return ;if (bdi_write_congested (bdi))return ;
fs/gfs2/ops_address.c +12 -0
@@ -372,6 +372,18 @@ retry: PAGECACHE_TAG_DIRTY, min(end - index, (pgoff_t)PAGEVEC_SIZE-1) + 1))) { scanned =1 ; + +/* +* If io group page belongs to is congested. bail out. +*/ +if (wbc->nonblocking + &&bdi_write_congested_group (bdi, pvec.pages[0 ])) { + wbc->encountered_congestion =1 ; + done =1 ; +pagevec_release (&pvec); +break ; + } + ret =gfs2_write_jdata_pagevec (mapping, wbc, &pvec, nr_pages, end);if (ret) done =1 ;
fs/nilfs2/segbuf.c +2 -1
@@ -267,8 +267,9 @@ static int nilfs_submit_seg_bio(struct nilfs_write_info *wi, int mode) {struct bio *bio = wi->bio;int err; +struct page *page =bio_iovec_idx (bio,0 )->bv_page; -if (wi->nbio >0 &&bdi_write_congested (wi->bdi)) { +if (wi->nbio >0 &&bdi_or_group_write_congested (wi->bdi, page)) {wait_for_completion (&wi->bio_event); wi->nbio--;if (unlikely (atomic_read (&wi->err))) {
fs/xfs/linux-2.6/xfs_aops.c +1 -1
@@ -891,7 +891,7 @@ xfs_convert_page( bdi = inode->i_mapping->backing_dev_info; wbc->nr_to_write--; -if (bdi_write_congested (bdi)) { +if (bdi_or_group_write_congested (bdi, page)) { wbc->encountered_congestion =1 ; done =1 ; }else if (wbc->nr_to_write <=0 ) {
fs/xfs/linux-2.6/xfs_buf.c +1 -1
@@ -714,7 +714,7 @@ xfs_buf_readahead(struct backing_dev_info *bdi; bdi = target->bt_mapping->backing_dev_info; -if (bdi_read_congested (bdi)) +if (bdi_or_group_read_congested (bdi,NULL ))return ; flags |= (XBF_TRYLOCK |XBF_ASYNC |XBF_READ_AHEAD );
include/linux/backing-dev.h +59 -2
79 lines not on this page · raw message
include/linux/biotrack.h +6 -0
@@ -49,6 +49,7 @@ extern void blkio_cgroup_copy_owner(struct page *page, struct page *opage);extern struct io_context *get_blkio_cgroup_iocontext (struct bio *bio);extern unsigned long get_blkio_cgroup_id (struct bio *bio); +extern unsigned long get_blkio_cgroup_id_page (struct page *page);extern struct cgroup *blkio_cgroup_lookup (int id);#else /* CONFIG_CGROUP_BIO */
@@ -92,6 +93,11 @@ static inline unsigned long get_blkio_cgroup_id(struct bio *bio)return 0 ; } +static inline unsigned long get_blkio_cgroup_id_page (struct page *page) +{ +return 0 ; +} +#endif /* CONFIG_CGROUP_BLKIO */ #endif /* _LINUX_BIOTRACK_H */
include/linux/blkdev.h +5 -0
@@ -880,6 +880,11 @@ static inline void blk_set_queue_congested(struct request_queue *q, int rw)set_bdi_congested (&q->backing_dev_info, rw); } +#ifdef CONFIG_GROUP_IOSCHED +extern int blk_queue_io_group_congested (struct backing_dev_info *bdi, +int bdi_bits,struct page *page); +#endif +extern void blk_start_queue (struct request_queue *q);extern void blk_stop_queue (struct request_queue *q);extern void blk_sync_queue (struct request_queue *q);
mm/backing-dev.c +62 -0
71 lines not on this page · raw message
mm/biotrack.c +21 -0
@@ -212,6 +212,27 @@ unsigned long get_blkio_cgroup_id(struct bio *bio) }/** +* get_blkio_cgroup_id_page() - determine the blkio-cgroup ID +* @page: the &struct page which describes the I/O +* +* Returns the blkio-cgroup ID of a given page. A return value zero +* means that the page associated with the IO belongs to default_blkio_cgroup. +*/ +unsigned long get_blkio_cgroup_id_page (struct page *page) +{ +struct page_cgroup *pc; +unsigned long id =0 ; + + pc =lookup_page_cgroup (page); +if (pc) { +lock_page_cgroup (pc); + id =page_cgroup_get_id (pc); +unlock_page_cgroup (pc); + } +return id; +} + +/** * get_blkio_cgroup_iocontext() - determine the blkio-cgroup iocontext * @bio: the &struct bio which describe the I/O *
mm/page-writeback.c +11 -0
@@ -981,6 +981,17 @@ retry:if (nr_pages ==0 )break ; +/* +* If the io group page will go into is congested, bail out. +*/ +if (wbc->nonblocking + &&bdi_write_congested_group (bdi, pvec.pages[0 ])) { + wbc->encountered_congestion =1 ; + done =1 ; +pagevec_release (&pvec); +break ; + } +for (i =0 ; i < nr_pages; i++) {struct page *page = pvec.pages[i];
mm/readahead.c +2 -2
@@ -240,7 +240,7 @@ int force_page_cache_readahead(struct address_space *mapping, struct file *filp,int do_page_cache_readahead (struct address_space *mapping,struct file *filp,pgoff_t offset,unsigned long nr_to_read) { -if (bdi_read_congested (mapping->backing_dev_info)) +if (bdi_or_group_read_congested (mapping->backing_dev_info,NULL ))return -1 ;return __do_page_cache_readahead (mapping, filp, offset, nr_to_read,0 );
@@ -485,7 +485,7 @@ page_cache_async_readahead(struct address_space *mapping,/* * Defer asynchronous read-ahead on IO congestion. */ -if (bdi_read_congested (mapping->backing_dev_info)) +if (bdi_or_group_read_congested (mapping->backing_dev_info,NULL ))return ;/* do read-ahead */
Signature · 1 line
1.6.0.6
PATCH 18/20 io-controller: Support per cgroup per device weights and io class
In reply to a message not in the archiveThis patch enables per-cgroup per-device weight and ioprio_class handling. A new cgroup interface "policy" is introduced. You can make use of this file to configure weight and ioprio_class for each device in a given cgroup. The original "weight" and "ioprio_class" files are still available. If you don't do special configuration for a particular device, "weight" and "ioprio_class" are used as default values in this device.
You can use the following format to play with the new interface. #echo DEV:weight:ioprio_class > /patch/to/cgroup/policy weight=0 means removing the policy for DEV.
Examples:
Configure weight=300 ioprio_class=2 on /dev/hdb in this cgroup
# echo /dev/hdb:300:2 > io.policy # cat io.policy dev weight class /dev/hdb 300 2
Configure weight=500 ioprio_class=1 on /dev/hda in this cgroup
# echo /dev/hda:500:1 > io.policy # cat io.policy dev weight class /dev/hda 500 1 /dev/hdb 300 2
Remove the policy for /dev/hda in this cgroup
# echo /dev/hda:0:1 > io.policy # cat io.policy dev weight class /dev/hdb 300 2
Changelog (v1 -> v2)
- Rename some structures
- Use spin_lock_irqsave() and spin_lock_irqrestore() version to prevent from enabling the interrupts unconditionally.
- Fix policy setup bug when switching to another io scheduler.
- If a policy is available for a specific device, don't update weight and io class when writing "weight" and "iprio_class".
- Fix a bug when parsing policy string.
--- block/elevator-fq.c | 243 ++++++++++++++++++++++++++++++++++++++++++++++++++- block/elevator-fq.h | 11 +++ 2 files changed, 250 insertions(+), 4 deletions(-)
block/elevator-fq.c +239 -4
@@ -15,6 +15,7 @@#include <linux/blktrace_api.h> #include <linux/seq_file.h> #include <linux/biotrack.h> +#include <linux/genhd.h> /* Values taken from cfq */ const int elv_slice_sync =HZ /10 ;
@@ -1168,12 +1169,31 @@ struct io_group *io_cgroup_lookup_group(struct io_cgroup *iocg, void *key)return NULL ; } -void io_group_init_entity (struct io_cgroup *iocg,struct io_group *iog) +static struct io_policy_node *policy_search_node (const struct io_cgroup *iocg, +dev_t dev); + +void io_group_init_entity (struct io_cgroup *iocg,struct io_group *iog, +dev_t dev) {struct io_entity *entity = &iog->entity; +struct io_policy_node *pn; +unsigned long flags; + +spin_lock_irqsave (&iocg->lock, flags); + pn =policy_search_node (iocg, dev); +if (pn) { + entity->weight = pn->weight; + entity->new_weight = pn->weight; + entity->ioprio_class = pn->ioprio_class; + entity->new_ioprio_class = pn->ioprio_class; + }else { + entity->weight = iocg->weight; + entity->new_weight = iocg->weight; + entity->ioprio_class = iocg->ioprio_class; + entity->new_ioprio_class = iocg->ioprio_class; + } +spin_unlock_irqrestore (&iocg->lock, flags); - entity->weight = entity->new_weight = iocg->weight; - entity->ioprio_class = entity->new_ioprio_class = iocg->ioprio_class; entity->ioprio_changed =1 ; entity->my_sched_data = &iog->sched_data; }
@@ -1225,6 +1245,7 @@ static int io_cgroup_##__VAR##_write(struct cgroup *cgroup, \struct io_cgroup *iocg; \struct io_group *iog; \struct hlist_node *n; \ +struct io_policy_node *pn; \ \if (val < (__MIN) || val > (__MAX)) \return -EINVAL ; \
@@ -1237,6 +1258,9 @@ static int io_cgroup_##__VAR##_write(struct cgroup *cgroup, \spin_lock_irq (&iocg->lock); \ iocg->__VAR = (unsigned long )val; \ hlist_for_each_entry(iog, n, &iocg->group_data, group_node) { \ + pn =policy_search_node (iocg, iog->dev); \ +if (pn) \ +continue ; \ iog->entity.new_##__VAR = (unsigned long)val; \smp_wmb (); \ iog->entity.ioprio_changed =1 ; \
@@ -1352,7 +1376,7 @@ struct io_group *io_group_chain_alloc(struct request_queue *q, void *key,sscanf (dev_name (bdi->dev),"%u:%u" , &major, &minor); iog->dev =MKDEV (major, minor); -io_group_init_entity (iocg, iog); +io_group_init_entity (iocg, iog, iog->dev); iog->my_entity = &iog->entity;atomic_set (&iog->ref,0 );
@@ -1665,8 +1689,212 @@ struct io_group *io_alloc_root_group(struct request_queue *q,return iog; } +static int io_cgroup_policy_read (struct cgroup *cgrp,struct cftype *cft, +struct seq_file *m) +{ +struct io_cgroup *iocg; +struct io_policy_node *pn; + + iocg =cgroup_to_io_cgroup (cgrp); + +if (list_empty (&iocg->policy_list)) + goto out; + +seq_printf (m,"dev weight class\n" ); + +spin_lock_irq (&iocg->lock); +list_for_each_entry (pn, &iocg->policy_list, node) { +seq_printf (m,"%s %lu %lu\n" , pn->dev_name, + pn->weight, pn->ioprio_class); + } +spin_unlock_irq (&iocg->lock); +out: +return 0 ; +} + +static inline void policy_insert_node (struct io_cgroup *iocg, +struct io_policy_node *pn) +{ +list_add (&pn->node, &iocg->policy_list); +} + +/* Must be called with iocg->lock held */ +static inline void policy_delete_node (struct io_policy_node *pn) +{ +list_del (&pn->node); +} + +/* Must be called with iocg->lock held */ +static struct io_policy_node *policy_search_node (const struct io_cgroup *iocg, +dev_t dev) +{ +struct io_policy_node *pn; + +if (list_empty (&iocg->policy_list)) +return NULL ; + +list_for_each_entry (pn, &iocg->policy_list, node) { +if (pn->dev == dev) +return pn; + } + +return NULL ; +} + +static int devname_to_devnum (const char *buf,dev_t *dev) +{ +struct block_device *bdev; +struct gendisk *disk; +int part; + + bdev =lookup_bdev (buf); +if (IS_ERR (bdev)) +return -ENODEV ; + + disk =get_gendisk (bdev->bd_dev, &part); +if (part) +return -EINVAL ; + + *dev =MKDEV (disk->major, disk->first_minor); +bdput (bdev); + +return 0 ; +} + +static int policy_parse_and_set (char *buf,struct io_policy_node *newpn) +{ +char *s[3 ], *p; +int ret; +int i =0 ; + +memset (s,0 ,sizeof (s)); +while ((p =strsep (&buf,":" )) !=NULL ) { +if (!*p) +continue ; + s[i++] = p; + } + + ret =devname_to_devnum (s[0 ], &newpn->dev); +if (ret) +return ret; + +strcpy (newpn->dev_name, s[0 ]); + +if (s[1 ] ==NULL ) +return -EINVAL ; + + ret =strict_strtoul (s[1 ],10 , &newpn->weight); +if (ret || newpn->weight >WEIGHT_MAX ) +return -EINVAL ; + +if (s[2 ] ==NULL ) +return -EINVAL ; + + ret =strict_strtoul (s[2 ],10 , &newpn->ioprio_class); +if (ret || newpn->ioprio_class <IOPRIO_CLASS_RT || + newpn->ioprio_class >IOPRIO_CLASS_IDLE ) +return -EINVAL ; + +return 0 ; +} + +static int io_cgroup_policy_write (struct cgroup *cgrp,struct cftype *cft, +const char *buffer) +{ +struct io_cgroup *iocg; +struct io_policy_node *newpn, *pn; +char *buf; +int ret =0 ; +int keep_newpn =0 ; +struct hlist_node *n; +struct io_group *iog; + + buf =kstrdup (buffer,GFP_KERNEL ); +if (!buf) +return -ENOMEM ; + + newpn =kzalloc (sizeof (*newpn),GFP_KERNEL ); +if (!newpn) { + ret = -ENOMEM ; + goto free_buf; + } + + ret =policy_parse_and_set (buf, newpn); +if (ret) + goto free_newpn; + +if (!cgroup_lock_live_group (cgrp)) { + ret = -ENODEV ; + goto free_newpn; + } + + iocg =cgroup_to_io_cgroup (cgrp); +spin_lock_irq (&iocg->lock); + + pn =policy_search_node (iocg, newpn->dev); +if (!pn) { +if (newpn->weight !=0 ) { +policy_insert_node (iocg, newpn); + keep_newpn =1 ; + } + goto update_io_group; + } + +if (newpn->weight ==0 ) { +/* weight == 0 means deleteing a policy */ +policy_delete_node (pn); + goto update_io_group; + } + + pn->weight = newpn->weight; + pn->ioprio_class = newpn->ioprio_class; + +update_io_group: +hlist_for_each_entry (iog, n, &iocg->group_data, group_node) { +if (iog->dev == newpn->dev) { +if (newpn->weight) { + iog->entity.new_weight = newpn->weight; + iog->entity.new_ioprio_class = + newpn->ioprio_class; +/* +* iog weight and ioprio_class updating +* actually happens if ioprio_changed is set. +* So ensure ioprio_changed is not set until +* new weight and new ioprio_class are updated. +*/ +smp_wmb (); + iog->entity.ioprio_changed =1 ; + }else { + iog->entity.new_weight = iocg->weight; + iog->entity.new_ioprio_class = + iocg->ioprio_class; + +/* The same as above */ +smp_wmb (); + iog->entity.ioprio_changed =1 ; + } + } + } +spin_unlock_irq (&iocg->lock); + +cgroup_unlock (); + +free_newpn: +if (!keep_newpn) +kfree (newpn); +free_buf: +kfree (buf); +return ret; +} +struct cftype bfqio_files[] = { { + .name ="policy" , + .read_seq_string = io_cgroup_policy_read, + .write_string = io_cgroup_policy_write, + .max_write_len =256 , + }, + { .name ="weight" , .read_u64 = io_cgroup_weight_read, .write_u64 = io_cgroup_weight_write,
@@ -1708,6 +1936,7 @@ struct cgroup_subsys_state *iocg_create(struct cgroup_subsys *subsys,INIT_HLIST_HEAD (&iocg->group_data); iocg->weight =IO_DEFAULT_GRP_WEIGHT ; iocg->ioprio_class =IO_DEFAULT_GRP_CLASS ; +INIT_LIST_HEAD (&iocg->policy_list);return &iocg->css; }
@@ -1911,6 +2140,7 @@ void iocg_destroy(struct cgroup_subsys *subsys, struct cgroup *cgroup)struct io_group *iog;struct elv_fq_data *efqd;unsigned long uninitialized_var (flags ); +struct io_policy_node *pn, *pntmp;/* * io groups are linked in two lists. One list is maintained
@@ -1949,6 +2179,11 @@ remove_entry: goto remove_entry; done: +list_for_each_entry_safe (pn, pntmp, &iocg->policy_list, node) { +policy_delete_node (pn); +kfree (pn); + } +free_css_id (&io_subsys, &iocg->css);rcu_read_unlock ();BUG_ON (!hlist_empty (&iocg->group_data));
block/elevator-fq.h +11 -0
@@ -267,6 +267,14 @@ struct io_group {struct rcu_head rcu_head; }; +struct io_policy_node { +struct list_head node; +char dev_name[32 ]; +dev_t dev; +unsigned long weight; +unsigned long ioprio_class; +}; +/** * struct bfqio_cgroup - bfq cgroup data structure. * @css: subsystem state for bfq in the containing cgroup.
@@ -283,6 +291,9 @@ struct io_cgroup {unsigned long weight, ioprio_class; +/* list of io_policy_node */ +struct list_head policy_list; +spinlock_t lock;struct hlist_head group_data; };
Signature · 1 line
1.6.0.6
PATCH 19/20 io-controller: Debug hierarchical IO scheduling
In reply to a message not in the archiveo Littile debugging aid for hierarchical IO scheduling.
o Enabled under CONFIG_DEBUG_GROUP_IOSCHED
o Currently it outputs more debug messages in blktrace output which helps a great deal in debugging in hierarchical setup. It also creates additional cgroup interfaces io.disk_queue and io.disk_dequeue to output some more debugging data.
--- block/Kconfig.iosched | 10 ++- block/as-iosched.c | 50 ++++++--- block/elevator-fq.c | 277 ++++++++++++++++++++++++++++++++++++++++++++++++- block/elevator-fq.h | 36 +++++++ 4 files changed, 351 insertions(+), 22 deletions(-)
block/Kconfig.iosched +9 -1
@@ -140,6 +140,14 @@ config TRACK_ASYNC_CONTEXT request, original owner of the bio is decided by using io tracking patches otherwise we continue to attribute the request to the submitting thread. -endmenu +config DEBUG_GROUP_IOSCHED + bool "Debug Hierarchical Scheduling support" + depends on CGROUPS && GROUP_IOSCHED + default n + ---help--- + Enable some debugging hooks for hierarchical scheduling support. + Currently it just outputs more information in blktrace output. + +endmenu endif
block/as-iosched.c +32 -18
152 lines not on this page · raw message
block/elevator-fq.c +274 -3
382 lines not on this page · raw message
block/elevator-fq.h +36 -0
@@ -265,6 +265,23 @@ struct io_group {/* request list associated with the group */ struct request_list rl;struct rcu_head rcu_head; + +#ifdef CONFIG_DEBUG_GROUP_IOSCHED +/* How many times this group has been added to active tree */ +unsigned long queue; + +/* How long this group remained on active tree, in ms */ +unsigned long queue_duration; + +/* When was this group added to active tree */ +unsigned long queue_start; + +/* How many times this group has been removed from active tree */ +unsigned long dequeue; + +/* Store cgroup path */ +char path[128 ]; +#endif };struct io_policy_node {
@@ -367,10 +384,29 @@ extern int elv_slice_idle;extern int elv_slice_async;/* Logging facilities. */ +#ifdef CONFIG_DEBUG_GROUP_IOSCHED +#define elv_log_ioq(efqd, ioq, fmt, args...) \ +{ \ + blk_add_trace_msg((efqd)->queue, "elv%d%c %s " fmt, (ioq)->pid, \ + elv_ioq_sync(ioq) ? 'S' : 'A', \ + ioq_to_io_group(ioq)->path, ##args); \ +} + +#define elv_log_iog(efqd, iog, fmt, args...) \ +{ \ + blk_add_trace_msg((efqd)->queue, "elv %s " fmt, (iog)->path, ##args); \ +} + +#else #define elv_log_ioq(efqd, ioq, fmt, args...) \ blk_add_trace_msg((efqd)->queue, "elv%d%c " fmt, (ioq)->pid, \ elv_ioq_sync(ioq) ? 'S' : 'A', ##args) +#define elv_log_iog(efqd, iog, fmt, args...) \ + blk_add_trace_msg((efqd)->queue, "elv " fmt, ##args) + +#endif + #define elv_log(efqd, fmt, args...) \ blk_add_trace_msg((efqd)->queue, "elv " fmt, ##args)
Signature · 1 line
1.6.0.6
PATCH 20/20 io-controller: experimental debug patch for async queue wait before expiry
In reply to a message not in the archiveo A debug patch which does wait for next IO from async queue once it becomes empty.
o For async writes, traffic seen by IO scheduler is not in proportion to the weight of the cgroup task/page belongs to. So if there are two processes doing heavy writeouts in two cgroups with weights 1000 and 500 respectively, then IO scheduler does not see more traffic/IO from higher weight cgroup even if IO scheduler tries to give it higher disk time. Effectively, the async queue belonging to higher weight cgroup becomes empty, and gets out of contention for disk and lower weight cgroup gets to use disk giving an impression in user space that higher weight cgroup did not get higher time to disk.
o This is more of a problem at page cache level where a higher weight process might be writing out the pages of lower weight process etc and should be fixed there.
o While we fix those issues, introducing this debug patch which allows one
to idle on async queue (tunable via /sys/blolc/<disk>/queue/async_slice_idle) so that once a higher weight queue becomes empty, instead of expiring it we try to wait for next request to come from that queue hence giving it
higher disk time. A higher value of async_slice_idle, around 300ms, helps
me get some right numbers for my setup. Note: higher disk time would not
necessarily translate in more IO done as higher weight group is not pushing
enough IO to io scheduler. It is just a debugging aid to prove correctness
of IO controller by providing higher disk times to higher weight cgroup.
--- block/cfq-iosched.c | 1 + block/elevator-fq.c | 43 +++++++++++++++++++++++++++++++++++++++---- block/elevator-fq.h | 5 +++++ 3 files changed, 45 insertions(+), 4 deletions(-)
block/cfq-iosched.c +1 -0
@@ -2093,6 +2093,7 @@ static struct elv_fs_entry cfq_attrs[] = {ELV_ATTR (slice_sync),ELV_ATTR (slice_async),ELV_ATTR (fairness), +ELV_ATTR (async_slice_idle), __ATTR_NULL };
block/elevator-fq.c +39 -4
103 lines not on this page · raw message
block/elevator-fq.h +5 -0
@@ -359,6 +359,8 @@ struct elv_fq_data {* users of this functionality. */ unsigned int elv_slice_idle; +/* idle slice for async queue */ +unsigned int elv_async_slice_idle;struct timer_list idle_slice_timer;struct work_struct unplug_work;
@@ -685,6 +687,9 @@ extern ssize_t elv_slice_async_store(struct elevator_queue *q, const char *name,extern ssize_t elv_fairness_show (struct elevator_queue *q,char *name);extern ssize_t elv_fairness_store (struct elevator_queue *q,const char *name,size_t count); +extern ssize_t elv_async_slice_idle_show (struct elevator_queue *q,char *name); +extern ssize_t elv_async_slice_idle_store (struct elevator_queue *q, +const char *name,size_t count);/* Functions used by elevator.c */ extern int elv_init_fq_data (struct request_queue *q,struct elevator_queue *e);
Signature · 1 line
1.6.0.6
Quoted 4 lines · Vivek Goyal
Hi All,
Here is the V5 of the IO controller patches generated on top of 2.6.30.
[snip]
Quoted 3 lines
Testing
=======
[snip]
I've not been reading through the discussions in complete detail, but I see no reference to async reads or aio. In the case of aio, aio presumes the context of the user space process. Could you elaborate on any testing you've done with these cases?
Signature · 1 line
Balbir
Hi,
Quoted 13 lines · Vivek Goyal, on 06/20/2009 12:37 AM wrote:
Hi All,
Here is the V5 of the IO controller patches generated on top of 2.6.30.
Previous versions of the patches was posted here.
(V1) http://lkml.org/lkml/2009/3/11/486
(V2) http://lkml.org/lkml/2009/5/5/275
(V3) http://lkml.org/lkml/2009/5/26/472
(V4) http://lkml.org/lkml/2009/6/8/580This patchset is still work in progress but I want to keep on getting the snapshot of my tree out at regular intervals to get the feedback hence V5.
[..]
Quoted 10 lines
Testing
=======I have been able to do only very basic testing of reads and writes.
Test1 (Fairness for synchronous reads)
======================================
- Two dd in two cgroups with cgrop weights 1000 and 500. Ran two "dd" in those cgroups (With CFQ scheduler and /sys/block/<device>/queue/fairness = 1)
Quoted 5 lines
dd if=/mnt/$BLOCKDEV/zerofile1 of=/dev/null &
dd if=/mnt/$BLOCKDEV/zerofile2 of=/dev/null &234179072 bytes (234 MB) copied, 3.9065 s, 59.9 MB/s 234179072 bytes (234 MB) copied, 5.19232 s, 45.1 MB/s
Sorry, but the above isn't a correct way to test proportional fairness for synchronous reads. You need throughput only when *both* dd's running, don't you?
Considering both transfers started simultaneously (which isn't obvious too) in the way you test the throughput value only for the first finished dd is correct, because after it finished, the second dd started transferring data *alone*, hence the result throughput value for it got partially for simultaneous, partially for alone reads, i.e. screwed.
I'd suggest you instead test as 2 runs of:
1. while true; do dd if=/mnt/$BLOCKDEV/zerofile1 of=/dev/null; done dd if=/mnt/$BLOCKDEV/zerofile2 of=/dev/null
2. while true; do dd if=/mnt/$BLOCKDEV/zerofile2 of=/dev/null; done dd if=/mnt/$BLOCKDEV/zerofile1 of=/dev/null
and take results from the standalone dd's.
Vlad
PATCH 2/20 io-controller: Common flat fair queuing code in elevaotor layer
In reply to Vivek GoyalThis is common fair queuing code in elevator layer. This is controlled by config option CONFIG_ELV_FAIR_QUEUING. This patch initially only introduces flat fair queuing support where there is only one group, "root group" and all the tasks belong to root group.
This elevator layer changes are backward compatible. That means any ioscheduler using old interfaces will continue to work.
This code is essentially the CFQ code for fair queuing. The primary difference is that flat rounding robin algorithm of CFQ has been replaced with BFQ (WF2Q+).
--- block/Kconfig.iosched | 13 + block/Makefile | 1 + block/elevator-fq.c | 2015 ++++++++++++++++++++++++++++++++++++++++++++++ block/elevator-fq.h | 473 +++++++++++ block/elevator.c | 46 +- include/linux/blkdev.h | 5 + include/linux/elevator.h | 51 ++ 7 files changed, 2593 insertions(+), 11 deletions(-) create mode 100644 block/elevator-fq.c create mode 100644 block/elevator-fq.h
block/Kconfig.iosched +13 -0
@@ -2,6 +2,19 @@ if BLOCK menu "IO Schedulers" +config ELV_FAIR_QUEUING + bool "Elevator Fair Queuing Support" + default n + ---help--- + Traditionally only cfq had notion of multiple queues and it did + fair queuing at its own. With the cgroups and need of controlling + IO, now even the simple io schedulers like noop, deadline, as will + have one queue per cgroup and will need hierarchical fair queuing. + Instead of every io scheduler implementing its own fair queuing + logic, this option enables fair queuing in elevator layer so that + other ioschedulers can make use of it. + If unsure, say N. + config IOSCHED_NOOP bool default y
block/Makefile +1 -0
@@ -15,3 +15,4 @@ obj-$(CONFIG_IOSCHED_CFQ) += cfq-iosched.o obj-$(CONFIG_BLOCK_COMPAT) += compat_ioctl.o obj-$(CONFIG_BLK_DEV_INTEGRITY) += blk-integrity.o +obj-$(CONFIG_ELV_FAIR_QUEUING) += elevator-fq.o
block/elevator-fq.c new file +2015 -0
2,015 lines not on this page · raw message
block/elevator-fq.h new file +473 -0
473 lines not on this page · raw message
block/elevator.c +35 -11
116 lines not on this page · raw message
include/linux/blkdev.h +5 -0
@@ -245,6 +245,11 @@ struct request {/* for bidi */ struct request *next_rq; + +#ifdef CONFIG_ELV_FAIR_QUEUING +/* io queue request belongs to */ +struct io_queue *ioq; +#endif };static inline unsigned short req_get_ioprio (struct request *req)
include/linux/elevator.h +51 -0
86 lines not on this page · raw message
Signature · 1 line
1.6.0.6
Re: [PATCH 02/20] io-controller: Common flat fair queuing code in elevaotor layer
Quoted 11 lines · Vivek Goyal
This is common fair queuing code in elevator layer. This is controlled by config option
CONFIG_ELV_FAIR_QUEUING. This patch initially only introduces flat fair queuing support where there is only one group, "root group" and all the tasks belong to root group.This elevator layer changes are backward compatible. That means any ioscheduler using old interfaces will continue to work.
This code is essentially the CFQ code for fair queuing. The primary difference is that flat rounding robin algorithm of CFQ has been replaced with BFQ (WF2Q+).
The patch is quite long and to be honest requires a long time to review, which I don't mind. I suspect my frequently diverted mind is likely to miss a lot in a big patch like this. Could you consider splitting this further if possible. I think you'll notice the number of reviews will also increase.
Quoted 82 lines
Signed-off-by Nauman Rafique <[email protected]>Signed-off-by Fabio Checconi <[email protected]>Signed-off-by Paolo Valente <[email protected]>Signed-off-by Aristeu Rozanski <[email protected]>Signed-off-by Gui Jianfeng <[email protected]>Signed-off-by Vivek Goyal <[email protected]>--- block/Kconfig.iosched | 13 + block/Makefile | 1 + block/elevator-fq.c | 2015 ++++++++++++++++++++++++++++++++++++++++++++++ block/elevator-fq.h | 473 +++++++++++ block/elevator.c | 46 +- include/linux/blkdev.h | 5 + include/linux/elevator.h | 51 ++ 7 files changed, 2593 insertions(+), 11 deletions(-) create mode 100644 block/elevator-fq.c create mode 100644 block/elevator-fq.hblock/Kconfig.iosched +13
-0@@ -2,6 +2,19 @@ if BLOCK menu "IO Schedulers" +config ELV_FAIR_QUEUING + bool "Elevator Fair Queuing Support" + default n + ---help--- + Traditionally only cfq had notion of multiple queues and it did + fair queuing at its own. With the cgroups and need of controlling + IO, now even the simple io schedulers like noop, deadline, as will + have one queue per cgroup and will need hierarchical fair queuing. + Instead of every io scheduler implementing its own fair queuing + logic, this option enables fair queuing in elevator layer so that + other ioschedulers can make use of it. + If unsure, say N. + config IOSCHED_NOOP bool default yblock/Makefile +1
-0@@ -15,3 +15,4 @@ obj-$(CONFIG_IOSCHED_CFQ) += cfq-iosched.o obj-$(CONFIG_BLOCK_COMPAT) += compat_ioctl.o obj-$(CONFIG_BLK_DEV_INTEGRITY) += blk-integrity.o +obj-$(CONFIG_ELV_FAIR_QUEUING) += elevator-fq.oblock/elevator-fq.c new file +25
-0@@ -0,0 +1,2015 @@ +/* +* BFQ: Hierarchical B-WF2Q+ scheduler. +* +* Based on ideas and code from CFQ: +* Copyright (C) 2003 Jens Axboe <[email protected]> +* +* Copyright (C) 2008 Fabio Checconi <[email protected]> +* Paolo Valente <[email protected]> +* Copyright (C) 2009 Vivek Goyal <[email protected]> +* Nauman Rafique <[email protected]> +*/ + +#include <linux/blkdev.h> +#include "elevator-fq.h" +#include <linux/blktrace_api.h> + +/* Values taken from cfq */ +const int elv_slice_sync =HZ /10 ; +int elv_slice_async =HZ /25 ; +const int elv_slice_async_rq =2 ; +int elv_slice_idle =HZ /125 ; +static struct kmem_cache *elv_ioq_pool; + +#define ELV_SLICE_SCALE (5) +#define ELV_HW_QUEUE_MIN (5)
block/elevator-fq.c line 35
Quoted 10 lines
block/elevator-fq.c new file +10
-0+#define IO_SERVICE_TREE_INIT ((struct io_service_tree) \ + { RB_ROOT, RB_ROOT, NULL, NULL, 0, 0 }) + +static inline struct io_queue *elv_close_cooperator (struct request_queue *q, +struct io_queue *ioq,int probe); +struct io_entity *bfq_lookup_next_entity (struct io_sched_data *sd, +int extract); + +static inline int elv_prio_slice (struct elv_fq_data *efqd,int sync, +unsigned short prio)
Why is the return type int and not unsigned int or unsigned long? Can the return value ever be negative?
Quoted 26 lines
block/elevator-fq.c new file +26
-0+{ +const int base_slice = efqd->elv_slice[sync]; + +WARN_ON (prio >=IOPRIO_BE_NR ); + +return base_slice + (base_slice/ELV_SLICE_SCALE * (4 - prio)); +} + +static inline int +elv_prio_to_slice (struct elv_fq_data *efqd,struct io_queue *ioq) +{ +return elv_prio_slice (efqd,elv_ioq_sync (ioq), ioq->entity.ioprio); +} + +/* Mainly the BFQ scheduling code Follows */ + +/* +* Shift for timestamp calculations. This actually limits the maximum +* service allowed in one timestamp delta (small shift values increase it), +* the maximum total weight that can be used for the queues in the system +* (big shift values increase it), and the period of virtual time wraparounds. +*/ +#define WFQ_SERVICE_SHIFT 22 + +/** +* bfq_gt - compare two timestamps.
block/elevator-fq.c line 71
Quoted 10 lines
block/elevator-fq.c new file +10
-0+* @a: first ts. +* @b: second ts. +* +* Return @a > @b, dealing with wrapping correctly. +*/ +static inline int bfq_gt (bfq_timestamp_t a,bfq_timestamp_t b) +{ +return (s64)(a - b) >0 ; +} +
a and b are of type u64, but cast to s64 to deal with wrapping? Correct?
block/elevator-fq.c line 81
Quoted 10 lines
block/elevator-fq.c new file +10
-0+/** +* bfq_delta - map service into the virtual time domain. +* @service: amount of service. +* @weight: scale factor. +*/ +static inline bfq_timestamp_t bfq_delta (bfq_service_t service, +bfq_weight_t weight) +{ +bfq_timestamp_t d = (bfq_timestamp_t )service <<WFQ_SERVICE_SHIFT ; +
Why is the case required? Does the compiler complain? service is already of the correct type.
block/elevator-fq.c line 82
Quoted 1 line
block/elevator-fq.c new file +1
-0+do_div (d, weight);
On a 64 system both d and weight are 64 bit, but on a 32 bit system weight is 32 bits. do_div expects a 64 bit dividend and 32 bit divisor
- no?
Quoted 5 lines
block/elevator-fq.c new file +5
-0+return d; +} + +/** +* bfq_calc_finish - assign the finish time to an entity.
block/elevator-fq.c line 97
Quoted 10 lines
block/elevator-fq.c new file +10
-0+* @entity: the entity to act upon. +* @service: the service to be charged to the entity. +*/ +static inline void bfq_calc_finish (struct io_entity *entity, +bfq_service_t service) +{ +BUG_ON (entity->weight ==0 ); + + entity->finish = entity->start +bfq_delta (service, entity->weight); +}
Should we BUG_ON (entity->finish == entity->start)? Or is that expected when the entity has no service time left.
Quoted 27 lines
block/elevator-fq.c new file +27
-0+ +static inline struct io_queue *io_entity_to_ioq (struct io_entity *entity) +{ +struct io_queue *ioq =NULL ; + +BUG_ON (entity ==NULL ); +if (entity->my_sched_data ==NULL ) + ioq = container_of(entity, struct io_queue, entity); +return ioq; +} + +/** +* bfq_entity_of - get an entity from a node. +* @node: the node field of the entity. +* +* Convert a node pointer to the relative entity. This is used only +* to simplify the logic of some functions and not as the generic +* conversion mechanism because, e.g., in the tree walking functions, +* the check for a %NULL value would be redundant. +*/ +static inline struct io_entity *bfq_entity_of (struct rb_node *node) +{ +struct io_entity *entity =NULL ; + +if (node !=NULL ) + entity = rb_entry(node, struct io_entity, rb_node); +
block/elevator-fq.c line 134
Quoted 10 lines
block/elevator-fq.c new file +10
-0+return entity; +} + +/** +* bfq_extract - remove an entity from a tree. +* @root: the tree root. +* @entity: the entity to remove. +*/ +static inline void bfq_extract (struct rb_root *root,struct io_entity *entity) +{
Extract is not common terminology, why not use bfq_remove()?
block/elevator-fq.c line 138
Quoted 4 lines
block/elevator-fq.c new file +4
-0+BUG_ON (entity->tree != root); + + entity->tree =NULL ; +rb_erase (&entity->rb_node, root);
Don't you want to make entity->tree = NULL after rb_erase?
Quoted 6 lines
block/elevator-fq.c new file +6
-0+} + +/** +* bfq_idle_extract - extract an entity from the idle tree. +* @st: the service tree of the owning @entity. +* @entity: the entity being removed.
block/elevator-fq.c line 154
Quoted 10 lines
block/elevator-fq.c new file +10
-0+*/ +static void bfq_idle_extract (struct io_service_tree *st, +struct io_entity *entity) +{ +struct rb_node *next; + +BUG_ON (entity->tree != &st->idle); + +if (entity == st->first_idle) { + next =rb_next (&entity->rb_node);
What happens if next is NULL?
block/elevator-fq.c line 159
Quoted 5 lines
block/elevator-fq.c new file +5
-0+ st->first_idle =bfq_entity_of (next); + } + +if (entity == st->last_idle) { + next =rb_prev (&entity->rb_node);
What happens if next is NULL?
Quoted 49 lines
block/elevator-fq.c new file +49
-0+ st->last_idle =bfq_entity_of (next); + } + +bfq_extract (&st->idle, entity); +} + +/** +* bfq_insert - generic tree insertion. +* @root: tree root. +* @entity: entity to insert. +* +* This is used for the idle and the active tree, since they are both +* ordered by finish time. +*/ +static void bfq_insert (struct rb_root *root,struct io_entity *entity) +{ +struct io_entity *entry; +struct rb_node **node = &root->rb_node; +struct rb_node *parent =NULL ; + +BUG_ON (entity->tree !=NULL ); + +while (*node !=NULL ) { + parent = *node; + entry = rb_entry(parent, struct io_entity, rb_node); + +if (bfq_gt (entry->finish, entity->finish)) + node = &parent->rb_left; +else + node = &parent->rb_right; + } + +rb_link_node (&entity->rb_node, parent, node); +rb_insert_color (&entity->rb_node, root); + + entity->tree = root; +} + +/** +* bfq_update_min - update the min_start field of a entity. +* @entity: the entity to update. +* @node: one of its children. +* +* This function is called when @entity may store an invalid value for +* min_start due to updates to the active tree. The function assumes +* that the subtree rooted at @node (which may be its left or its right +* child) has a valid min_start value. +*/ +static inline void bfq_update_min(struct io_entity *entity,
block/elevator-fq.c line 218
Quoted 10 lines
block/elevator-fq.c new file +10
-0+ struct rb_node *node) +{ +struct io_entity *child; + +if (node !=NULL ) { + child = rb_entry(node, struct io_entity, rb_node); +if (bfq_gt (entity->min_start, child->min_start)) + entity->min_start = child->min_start; + } +}
So.. we check to see if child's min_time is lesser than the root entities or node entities and set it to the minimum of the two? Can you use min_t here?
Quoted 7 lines
block/elevator-fq.c new file +7
-0+ +/** +* bfq_update_active_node - recalculate min_start. +* @node: the node to update. +* +* @node may have changed position or one of its children may have moved, +* this function updates its min_start value. The left and right subtrees
block/elevator-fq.c line 235
Quoted 10 lines
block/elevator-fq.c new file +10
-0+* are assumed to hold a correct min_start value. +*/ +static inline void bfq_update_active_node (struct rb_node *node) +{ + struct io_entity *entity = rb_entry(node, struct io_entity, rb_node); + + entity->min_start = entity->start; +bfq_update_min (entity, node->rb_right); +bfq_update_min (entity, node->rb_left); +}
I don't like this every much, we set the min_time twice, this can be easily optimized to look at both left and right child and pick the minimum.
Quoted 21 lines
block/elevator-fq.c new file +21
-0+ +/** +* bfq_update_active_tree - update min_start for the whole active tree. +* @node: the starting node. +* +* @node must be the deepest modified node after an update. This function +* updates its min_start using the values held by its children, assuming +* that they did not change, and then updates all the nodes that may have +* changed in the path to the root. The only nodes that may have changed +* are the ones in the path or their siblings. +*/ +static void bfq_update_active_tree (struct rb_node *node) +{ +struct rb_node *parent; + +up: +bfq_update_active_node (node); + + parent =rb_parent (node); +if (parent ==NULL ) +return ;
block/elevator-fq.c line 266
Quoted 10 lines
block/elevator-fq.c new file +10
-0+ +if (node == parent->rb_left && parent->rb_right !=NULL ) +bfq_update_active_node (parent->rb_right); +else if (parent->rb_left !=NULL ) +bfq_update_active_node (parent->rb_left); + + node = parent; + goto up; +} +
For these functions, take a look at the walk function in the group scheduler that does update_shares
Quoted 55 lines
block/elevator-fq.c new file +55
-0+/** +* bfq_active_insert - insert an entity in the active tree of its group/device. +* @st: the service tree of the entity. +* @entity: the entity being inserted. +* +* The active tree is ordered by finish time, but an extra key is kept +* per each node, containing the minimum value for the start times of +* its children (and the node itself), so it's possible to search for +* the eligible node with the lowest finish time in logarithmic time. +*/ +static void bfq_active_insert (struct io_service_tree *st, +struct io_entity *entity) +{ +struct rb_node *node = &entity->rb_node; + +bfq_insert (&st->active, entity); + +if (node->rb_left !=NULL ) + node = node->rb_left; +else if (node->rb_right !=NULL ) + node = node->rb_right; + +bfq_update_active_tree (node); +} + +/** +* bfq_ioprio_to_weight - calc a weight from an ioprio. +* @ioprio: the ioprio value to convert. +*/ +static bfq_weight_t bfq_ioprio_to_weight (int ioprio) +{ +WARN_ON (ioprio <0 || ioprio >=IOPRIO_BE_NR ); +return IOPRIO_BE_NR - ioprio; +} + +void bfq_get_entity (struct io_entity *entity) +{ +struct io_queue *ioq =io_entity_to_ioq (entity); + +if (ioq) +elv_get_ioq (ioq); +} + +void bfq_init_entity (struct io_entity *entity,struct io_group *iog) +{ + entity->ioprio = entity->new_ioprio; + entity->ioprio_class = entity->new_ioprio_class; + entity->sched_data = &iog->sched_data; +} + +/** +* bfq_find_deepest - find the deepest node that an extraction can modify. +* @node: the node being removed. +* +* Do the first step of an extraction in an rb tree, looking for the
block/elevator-fq.c line 331
Quoted 10 lines
block/elevator-fq.c new file +10
-0+* node that will replace @node, and returning the deepest node that +* the following modifications to the tree can touch. If @node is the +* last node in the tree return %NULL. +*/ +static struct rb_node *bfq_find_deepest (struct rb_node *node) +{ +struct rb_node *deepest; + +if (node->rb_right ==NULL && node->rb_left ==NULL ) + deepest =rb_parent (node);
Why is the parent the deepest? Shouldn't node be the deepest?
Quoted 4 lines
block/elevator-fq.c new file +4
-0+else if (node->rb_right ==NULL ) + deepest = node->rb_left; +else if (node->rb_left ==NULL ) + deepest = node->rb_right;
block/elevator-fq.c line 345
Quoted 10 lines
block/elevator-fq.c new file +10
-0+else { + deepest =rb_next (node); +if (deepest->rb_right !=NULL ) + deepest = deepest->rb_right; +else if (rb_parent (deepest) != node) + deepest =rb_parent (deepest); + } + +return deepest; +}
The function is not clear, could you please define deepest node better?
Quoted 8 lines
block/elevator-fq.c new file +8
-0+ +/** +* bfq_active_extract - remove an entity from the active tree. +* @st: the service_tree containing the tree. +* @entity: the entity being removed. +*/ +static void bfq_active_extract (struct io_service_tree *st, +struct io_entity *entity)
block/elevator-fq.c line 363
Quoted 10 lines
block/elevator-fq.c new file +10
-0+{ +struct rb_node *node; + + node =bfq_find_deepest (&entity->rb_node); +bfq_extract (&st->active, entity); + +if (node !=NULL ) +bfq_update_active_tree (node); +} +
Just to check my understanding, every time an active node is added/removed, we update the min_time of the entire tree.
Quoted 328 lines
block/elevator-fq.c new file +328
-0+/** +* bfq_idle_insert - insert an entity into the idle tree. +* @st: the service tree containing the tree. +* @entity: the entity to insert. +*/ +static void bfq_idle_insert (struct io_service_tree *st, +struct io_entity *entity) +{ +struct io_entity *first_idle = st->first_idle; +struct io_entity *last_idle = st->last_idle; + +if (first_idle ==NULL ||bfq_gt (first_idle->finish, entity->finish)) + st->first_idle = entity; +if (last_idle ==NULL ||bfq_gt (entity->finish, last_idle->finish)) + st->last_idle = entity; + +bfq_insert (&st->idle, entity); +} + +/** +* bfq_forget_entity - remove an entity from the wfq trees. +* @st: the service tree. +* @entity: the entity being removed. +* +* Update the device status and forget everything about @entity, putting +* the device reference to it, if it is a queue. Entities belonging to +* groups are not refcounted. +*/ +static void bfq_forget_entity (struct io_service_tree *st, +struct io_entity *entity) +{ +struct io_queue *ioq =NULL ; + +BUG_ON (!entity->on_st); + entity->on_st =0 ; + st->wsum -= entity->weight; + ioq =io_entity_to_ioq (entity); +if (!ioq) +return ; +elv_put_ioq (ioq); +} + +/** +* bfq_put_idle_entity - release the idle tree ref of an entity. +* @st: service tree for the entity. +* @entity: the entity being released. +*/ +void bfq_put_idle_entity (struct io_service_tree *st, +struct io_entity *entity) +{ +bfq_idle_extract (st, entity); +bfq_forget_entity (st, entity); +} + +/** +* bfq_forget_idle - update the idle tree if necessary. +* @st: the service tree to act upon. +* +* To preserve the global O(log N) complexity we only remove one entry here; +* as the idle tree will not grow indefinitely this can be done safely. +*/ +void bfq_forget_idle (struct io_service_tree *st) +{ +struct io_entity *first_idle = st->first_idle; +struct io_entity *last_idle = st->last_idle; + +if (RB_EMPTY_ROOT (&st->active) && last_idle !=NULL && + !bfq_gt (last_idle->finish, st->vtime)) { +/* +* Active tree is empty. Pull back vtime to finish time of +* last idle entity on idle tree. +* Rational seems to be that it reduces the possibility of +* vtime wraparound (bfq_gt(V-F) < 0). +*/ + st->vtime = last_idle->finish; + } + +if (first_idle !=NULL && !bfq_gt (first_idle->finish, st->vtime)) +bfq_put_idle_entity (st, first_idle); +} + + +static struct io_service_tree * +__bfq_entity_update_prio (struct io_service_tree *old_st, +struct io_entity *entity) +{ +struct io_service_tree *new_st = old_st; +struct io_queue *ioq =io_entity_to_ioq (entity); + +if (entity->ioprio_changed) { + entity->ioprio = entity->new_ioprio; + entity->ioprio_class = entity->new_ioprio_class; + entity->ioprio_changed =0 ; + +/* +* Also update the scaled budget for ioq. Group will get the +* updated budget once ioq is selected to run next. +*/ +if (ioq) { +struct elv_fq_data *efqd = ioq->efqd; + entity->budget =elv_prio_to_slice (efqd, ioq); + } + + old_st->wsum -= entity->weight; + entity->weight =bfq_ioprio_to_weight (entity->ioprio); + +/* +* NOTE: here we may be changing the weight too early, +* this will cause unfairness. The correct approach +* would have required additional complexity to defer +* weight changes to the proper time instants (i.e., +* when entity->finish <= old_st->vtime). +*/ + new_st =io_entity_service_tree (entity); + new_st->wsum += entity->weight; + +if (new_st != old_st) + entity->start = new_st->vtime; + } + +return new_st; +} + +/** +* __bfq_activate_entity - activate an entity. +* @entity: the entity being activated. +* +* Called whenever an entity is activated, i.e., it is not active and one +* of its children receives a new request, or has to be reactivated due to +* budget exhaustion. It uses the current budget of the entity (and the +* service received if @entity is active) of the queue to calculate its +* timestamps. +*/ +static void __bfq_activate_entity (struct io_entity *entity,int add_front) +{ +struct io_sched_data *sd = entity->sched_data; +struct io_service_tree *st =io_entity_service_tree (entity); + +if (entity == sd->active_entity) { +BUG_ON (entity->tree !=NULL ); +/* +* If we are requeueing the current entity we have +* to take care of not charging to it service it has +* not received. +*/ +bfq_calc_finish (entity, entity->service); + entity->start = entity->finish; + sd->active_entity =NULL ; + }else if (entity->tree == &st->active) { +/* +* Requeueing an entity due to a change of some +* next_active entity below it. We reuse the old +* start time. +*/ +bfq_active_extract (st, entity); + }else if (entity->tree == &st->idle) { +/* +* Must be on the idle tree, bfq_idle_extract() will +* check for that. +*/ +bfq_idle_extract (st, entity); + entity->start =bfq_gt (st->vtime, entity->finish) ? + st->vtime : entity->finish; + }else { +/* +* The finish time of the entity may be invalid, and +* it is in the past for sure, otherwise the queue +* would have been on the idle tree. +*/ + entity->start = st->vtime; + st->wsum += entity->weight; +bfq_get_entity (entity); + +BUG_ON (entity->on_st); + entity->on_st =1 ; + } + + st =__bfq_entity_update_prio (st, entity); +/* +* This is to emulate cfq like functionality where preemption can +* happen with-in same class, like sync queue preempting async queue +* May be this is not a very good idea from fairness point of view +* as preempting queue gains share. Keeping it for now. +*/ +if (add_front) { +struct io_entity *next_entity; + +/* +* Determine the entity which will be dispatched next +* Use sd->next_active once hierarchical patch is applied +*/ + next_entity =bfq_lookup_next_entity (sd,0 ); + +if (next_entity && next_entity != entity) { +struct io_service_tree *new_st; +bfq_timestamp_t delta; + + new_st =io_entity_service_tree (next_entity); + +/* +* At this point, both entities should belong to +* same service tree as cross service tree preemption +* is automatically taken care by algorithm +*/ +BUG_ON (new_st != st); + entity->finish = next_entity->finish -1 ; + delta =bfq_delta (entity->budget, entity->weight); + entity->start = entity->finish - delta; +if (bfq_gt (entity->start, st->vtime)) + entity->start = st->vtime; + } + }else { +bfq_calc_finish (entity, entity->budget); + } +bfq_active_insert (st, entity); +} + +/** +* bfq_activate_entity - activate an entity. +* @entity: the entity to activate. +*/ +void bfq_activate_entity (struct io_entity *entity,int add_front) +{ +__bfq_activate_entity (entity, add_front); +} + +/** +* __bfq_deactivate_entity - deactivate an entity from its service tree. +* @entity: the entity to deactivate. +* @requeue: if false, the entity will not be put into the idle tree. +* +* Deactivate an entity, independently from its previous state. If the +* entity was not on a service tree just return, otherwise if it is on +* any scheduler tree, extract it from that tree, and if necessary +* and if the caller did not specify @requeue, put it on the idle tree. +* +*/ +int __bfq_deactivate_entity (struct io_entity *entity,int requeue) +{ +struct io_sched_data *sd = entity->sched_data; +struct io_service_tree *st =io_entity_service_tree (entity); +int was_active = entity == sd->active_entity; +int ret =0 ; + +if (!entity->on_st) +return 0 ; + +BUG_ON (was_active && entity->tree !=NULL ); + +if (was_active) { +bfq_calc_finish (entity, entity->service); + sd->active_entity =NULL ; + }else if (entity->tree == &st->active) +bfq_active_extract (st, entity); +else if (entity->tree == &st->idle) +bfq_idle_extract (st, entity); +else if (entity->tree !=NULL ) +BUG (); + +if (!requeue || !bfq_gt (entity->finish, st->vtime)) +bfq_forget_entity (st, entity); +else +bfq_idle_insert (st, entity); + +BUG_ON (sd->active_entity == entity); + +return ret; +} + +/** +* bfq_deactivate_entity - deactivate an entity. +* @entity: the entity to deactivate. +* @requeue: true if the entity can be put on the idle tree +*/ +void bfq_deactivate_entity (struct io_entity *entity,int requeue) +{ +__bfq_deactivate_entity (entity, requeue); +} + +/** +* bfq_update_vtime - update vtime if necessary. +* @st: the service tree to act upon. +* +* If necessary update the service tree vtime to have at least one +* eligible entity, skipping to its start time. Assumes that the +* active tree of the device is not empty. +* +* NOTE: this hierarchical implementation updates vtimes quite often, +* we may end up with reactivated tasks getting timestamps after a +* vtime skip done because we needed a ->first_active entity on some +* intermediate node. +*/ +static void bfq_update_vtime (struct io_service_tree *st) +{ +struct io_entity *entry; +struct rb_node *node = st->active.rb_node; + + entry = rb_entry(node, struct io_entity, rb_node); +if (bfq_gt (entry->min_start, st->vtime)) { + st->vtime = entry->min_start; +bfq_forget_idle (st); + } +} + +/** +* bfq_first_active - find the eligible entity with the smallest finish time +* @st: the service tree to select from. +* +* This function searches the first schedulable entity, starting from the +* root of the tree and going on the left every time on this side there is +* a subtree with at least one eligible (start <= vtime) entity. The path +* on the right is followed only if a) the left subtree contains no eligible +* entities and b) no eligible entity has been found yet. +*/ +static struct io_entity *bfq_first_active_entity (struct io_service_tree *st) +{ +struct io_entity *entry, *first =NULL ; +struct rb_node *node = st->active.rb_node; + +while (node !=NULL ) { + entry = rb_entry(node, struct io_entity, rb_node); +left: +if (!bfq_gt (entry->start, st->vtime)) + first = entry; + +BUG_ON (bfq_gt (entry->min_start, st->vtime)); + +if (node->rb_left !=NULL ) {
block/elevator-fq.c line 701
Quoted 10 lines
block/elevator-fq.c new file +10
-0+ entry =rb_entry (node->rb_left, + struct io_entity, rb_node); +if (!bfq_gt (entry->min_start, st->vtime)) { + node = node->rb_left; + goto left; + } + } +if (first !=NULL ) +break ; + node = node->rb_right;
Please help me understand this, we sort the tree by finish time, but search by vtime, start_time. The worst case could easily be O(N), right?
Quoted 64 lines
block/elevator-fq.c new file +64
-0+ } + +BUG_ON (first ==NULL && !RB_EMPTY_ROOT (&st->active)); +return first; +} + +/** +* __bfq_lookup_next_entity - return the first eligible entity in @st. +* @st: the service tree. +* +* Update the virtual time in @st and return the first eligible entity +* it contains. +*/ +static struct io_entity *__bfq_lookup_next_entity (struct io_service_tree *st) +{ +struct io_entity *entity; + +if (RB_EMPTY_ROOT (&st->active)) +return NULL ; + +bfq_update_vtime (st); + entity =bfq_first_active_entity (st); +BUG_ON (bfq_gt (entity->start, st->vtime)); + +return entity; +} + +/** +* bfq_lookup_next_entity - return the first eligible entity in @sd. +* @sd: the sched_data. +* @extract: if true the returned entity will be also extracted from @sd. +* +* NOTE: since we cache the next_active entity at each level of the +* hierarchy, the complexity of the lookup can be decreased with +* absolutely no effort just returning the cached next_active value; +* we prefer to do full lookups to test the consistency of * the data +* structures. +*/ +struct io_entity *bfq_lookup_next_entity (struct io_sched_data *sd, +int extract) +{ +struct io_service_tree *st = sd->service_tree; +struct io_entity *entity; +int i; + +/* +* We should not call lookup when an entity is active, as doing lookup +* can result in an erroneous vtime jump. +*/ +BUG_ON (sd->active_entity !=NULL ); + +for (i =0 ; i <IO_IOPRIO_CLASSES ; i++, st++) { + entity =__bfq_lookup_next_entity (st); +if (entity !=NULL ) { +if (extract) { +bfq_active_extract (st, entity); + sd->active_entity = entity; + } +break ; + } + } + +return entity; +}
block/elevator-fq.c line 775
Quoted 10 lines
block/elevator-fq.c new file +10
-0+ +void entity_served (struct io_entity *entity,bfq_service_t served) +{ +struct io_service_tree *st; + + st =io_entity_service_tree (entity); + entity->service += served; +BUG_ON (st->wsum ==0 ); + st->vtime +=bfq_delta (served, st->wsum); +bfq_forget_idle (st);
Forget idle checks to see if the st->vtime > first_idle->finish, if so it pushes the first_idle to later, right?
Quoted 177 lines
block/elevator-fq.c new file +177
-0+} + +/** +* bfq_flush_idle_tree - deactivate any entity on the idle tree of @st. +* @st: the service tree being flushed. +*/ +void io_flush_idle_tree (struct io_service_tree *st) +{ +struct io_entity *entity = st->first_idle; + +for (; entity !=NULL ; entity = st->first_idle) +__bfq_deactivate_entity (entity,0 ); +} + +/* Elevator fair queuing function */ +struct io_queue *rq_ioq (struct request *rq) +{ +return rq->ioq; +} + +static inline struct io_queue *elv_active_ioq (struct elevator_queue *e) +{ +return e->efqd.active_queue; +} + +void *elv_active_sched_queue (struct elevator_queue *e) +{ +return ioq_sched_queue (elv_active_ioq (e)); +} +EXPORT_SYMBOL (elv_active_sched_queue); + +int elv_nr_busy_ioq (struct elevator_queue *e) +{ +return e->efqd.busy_queues; +} +EXPORT_SYMBOL (elv_nr_busy_ioq); + +int elv_hw_tag (struct elevator_queue *e) +{ +return e->efqd.hw_tag; +} +EXPORT_SYMBOL (elv_hw_tag); + +/* Helper functions for operating on elevator idle slice timer */ +int elv_mod_idle_slice_timer (struct elevator_queue *eq,unsigned long expires) +{ +struct elv_fq_data *efqd = &eq->efqd; + +return mod_timer (&efqd->idle_slice_timer, expires); +} +EXPORT_SYMBOL (elv_mod_idle_slice_timer); + +int elv_del_idle_slice_timer (struct elevator_queue *eq) +{ +struct elv_fq_data *efqd = &eq->efqd; + +return del_timer (&efqd->idle_slice_timer); +} +EXPORT_SYMBOL (elv_del_idle_slice_timer); + +unsigned int elv_get_slice_idle (struct elevator_queue *eq) +{ +return eq->efqd.elv_slice_idle; +} +EXPORT_SYMBOL (elv_get_slice_idle); + +void elv_ioq_served (struct io_queue *ioq,bfq_service_t served) +{ +entity_served (&ioq->entity, served); +} + +/* Tells whether ioq is queued in root group or not */ +static inline int is_root_group_ioq (struct request_queue *q, +struct io_queue *ioq) +{ +struct elv_fq_data *efqd = &q->elevator->efqd; + +return (ioq->entity.sched_data == &efqd->root_group->sched_data); +} + +/* +* sysfs parts below --> +*/ +static ssize_t +elv_var_show (unsigned int var,char *page) +{ +return sprintf (page,"%d\n" , var); +} + +static ssize_t +elv_var_store (unsigned int *var,const char *page,size_t count) +{ +char *p = (char *) page; + + *var =simple_strtoul (p, &p,10 ); +return count; +} + +#define SHOW_FUNCTION (__FUNC, __VAR, __CONV) \ +ssize_t __FUNC(struct elevator_queue *e, char *page) \ +{ \ + struct elv_fq_data *efqd = &e->efqd; \ + unsigned int __data = __VAR; \ + if (__CONV) \ + __data = jiffies_to_msecs(__data); \ + return elv_var_show(__data, (page)); \ +} +SHOW_FUNCTION (elv_slice_idle_show, efqd->elv_slice_idle,1 ); +EXPORT_SYMBOL (elv_slice_idle_show); +SHOW_FUNCTION (elv_slice_sync_show, efqd->elv_slice[1 ],1 ); +EXPORT_SYMBOL (elv_slice_sync_show); +SHOW_FUNCTION (elv_slice_async_show, efqd->elv_slice[0 ],1 ); +EXPORT_SYMBOL (elv_slice_async_show); +#undef SHOW_FUNCTION + +#define STORE_FUNCTION (__FUNC, __PTR,MIN ,MAX , __CONV) \ +ssize_t __FUNC(struct elevator_queue *e, const char *page, size_t count)\ +{ \ + struct elv_fq_data *efqd = &e->efqd; \ + unsigned int __data; \ + int ret = elv_var_store(&__data, (page), count); \ + if (__data < (MIN)) \ + __data = (MIN); \ + else if (__data > (MAX)) \ + __data = (MAX); \ + if (__CONV) \ + *(__PTR) = msecs_to_jiffies(__data); \ + else \ + *(__PTR) = __data; \ + return ret; \ +} +STORE_FUNCTION (elv_slice_idle_store, &efqd->elv_slice_idle,0 ,UINT_MAX ,1 ); +EXPORT_SYMBOL (elv_slice_idle_store); +STORE_FUNCTION (elv_slice_sync_store, &efqd->elv_slice[1 ],1 ,UINT_MAX ,1 ); +EXPORT_SYMBOL (elv_slice_sync_store); +STORE_FUNCTION (elv_slice_async_store, &efqd->elv_slice[0 ],1 ,UINT_MAX ,1 ); +EXPORT_SYMBOL (elv_slice_async_store); +#undef STORE_FUNCTION + +void elv_schedule_dispatch (struct request_queue *q) +{ +struct elv_fq_data *efqd = &q->elevator->efqd; + +if (elv_nr_busy_ioq (q->elevator)) { +elv_log (efqd,"schedule dispatch" ); +kblockd_schedule_work (efqd->queue, &efqd->unplug_work); + } +} +EXPORT_SYMBOL (elv_schedule_dispatch); + +void elv_kick_queue (struct work_struct *work) +{ +struct elv_fq_data *efqd = + container_of(work, struct elv_fq_data, unplug_work); +struct request_queue *q = efqd->queue; +unsigned long flags; + +spin_lock_irqsave (q->queue_lock, flags); +blk_start_queueing (q); +spin_unlock_irqrestore (q->queue_lock, flags); +} + +void elv_shutdown_timer_wq (struct elevator_queue *e) +{ +del_timer_sync (&e->efqd.idle_slice_timer); +cancel_work_sync (&e->efqd.unplug_work); +} +EXPORT_SYMBOL (elv_shutdown_timer_wq); + +void elv_ioq_set_prio_slice (struct request_queue *q,struct io_queue *ioq) +{ +struct elv_fq_data *efqd = &q->elevator->efqd; + + ioq->slice_end = jiffies + ioq->entity.budget; +elv_log_ioq (efqd, ioq,"set_slice=%lu" , ioq->entity.budget); +} +
block/elevator-fq.c line 962
Quoted 10 lines
block/elevator-fq.c new file +10
-0+static void elv_ioq_update_io_thinktime (struct io_queue *ioq) +{ +struct elv_fq_data *efqd = ioq->efqd; +unsigned long elapsed = jiffies - ioq->last_end_request; +unsigned long ttime =min (elapsed,2UL * efqd->elv_slice_idle); + + ioq->ttime_samples = (7 *ioq->ttime_samples +256 ) /8 ; + ioq->ttime_total = (7 *ioq->ttime_total +256 *ttime) /8 ; + ioq->ttime_mean = (ioq->ttime_total +128 ) / ioq->ttime_samples; +}
Not sure I understand the magical 7, 8, 2, 128 and 256. Please help me understand the algorithm.
Quoted 68 lines
block/elevator-fq.c new file +68
-0+ +/* +* Disable idle window if the process thinks too long. +* This idle flag can also be updated by io scheduler. +*/ +static void elv_ioq_update_idle_window (struct elevator_queue *eq, +struct io_queue *ioq,struct request *rq) +{ +int old_idle, enable_idle; +struct elv_fq_data *efqd = ioq->efqd; + +/* +* Don't idle for async or idle io prio class +*/ +if (!elv_ioq_sync (ioq) ||elv_ioq_class_idle (ioq)) +return ; + + enable_idle = old_idle =elv_ioq_idle_window (ioq); + +if (!efqd->elv_slice_idle) + enable_idle =0 ; +else if (ioq_sample_valid (ioq->ttime_samples)) { +if (ioq->ttime_mean > efqd->elv_slice_idle) + enable_idle =0 ; +else + enable_idle =1 ; + } + +/* +* From think time perspective idle should be enabled. Check with +* io scheduler if it wants to disable idling based on additional +* considrations like seek pattern. +*/ +if (enable_idle) { +if (eq->ops->elevator_update_idle_window_fn) + enable_idle = eq->ops->elevator_update_idle_window_fn ( + eq, ioq->sched_queue, rq); +if (!enable_idle) +elv_log_ioq (efqd, ioq,"iosched disabled idle" ); + } + +if (old_idle != enable_idle) { +elv_log_ioq (efqd, ioq,"idle=%d" , enable_idle); +if (enable_idle) +elv_mark_ioq_idle_window (ioq); +else +elv_clear_ioq_idle_window (ioq); + } +} + +struct io_queue *elv_alloc_ioq (struct request_queue *q,gfp_t gfp_mask) +{ +struct io_queue *ioq =NULL ; + + ioq =kmem_cache_alloc_node (elv_ioq_pool, gfp_mask, q->node); +return ioq; +} +EXPORT_SYMBOL (elv_alloc_ioq); + +void elv_free_ioq (struct io_queue *ioq) +{ +kmem_cache_free (elv_ioq_pool, ioq); +} +EXPORT_SYMBOL (elv_free_ioq); + +int elv_init_ioq (struct elevator_queue *eq,struct io_queue *ioq, +void *sched_queue,int ioprio_class,int ioprio, +int is_sync)
block/elevator-fq.c line 1040
Quoted 10 lines
block/elevator-fq.c new file +10
-0+{ +struct elv_fq_data *efqd = &eq->efqd; +struct io_group *iog =io_lookup_io_group_current (efqd->queue); + +RB_CLEAR_NODE (&ioq->entity.rb_node); +atomic_set (&ioq->ref,0 ); + ioq->efqd = efqd; +elv_ioq_set_ioprio_class (ioq, ioprio_class); +elv_ioq_set_ioprio (ioq, ioprio); + ioq->pid = current->pid;
Is pid used for cgroup association later? I don't see why we save the pid otherwise? If yes, why not store the cgroup of the current->pid?
Quoted 95 lines
block/elevator-fq.c new file +95
-0+ ioq->sched_queue = sched_queue; +if (is_sync && !elv_ioq_class_idle (ioq)) +elv_mark_ioq_idle_window (ioq); +bfq_init_entity (&ioq->entity, iog); + ioq->entity.budget =elv_prio_to_slice (efqd, ioq); +if (is_sync) + ioq->last_end_request = jiffies; + +return 0 ; +} +EXPORT_SYMBOL (elv_init_ioq); + +void elv_put_ioq (struct io_queue *ioq) +{ +struct elv_fq_data *efqd = ioq->efqd; + struct elevator_queue *e = container_of(efqd, struct elevator_queue, + efqd); + +BUG_ON (atomic_read (&ioq->ref) <=0 ); +if (!atomic_dec_and_test (&ioq->ref)) +return ; +BUG_ON (ioq->nr_queued); +BUG_ON (ioq->entity.tree !=NULL ); +BUG_ON (elv_ioq_busy (ioq)); +BUG_ON (efqd->active_queue == ioq); + +/* Can be called by outgoing elevator. Don't use q */ +BUG_ON (!e->ops->elevator_free_sched_queue_fn); + + e->ops->elevator_free_sched_queue_fn (e, ioq->sched_queue); +elv_log_ioq (efqd, ioq,"put_queue" ); +elv_free_ioq (ioq); +} +EXPORT_SYMBOL (elv_put_ioq); + +void elv_release_ioq (struct elevator_queue *e,struct io_queue **ioq_ptr) +{ +struct io_queue *ioq = *ioq_ptr; + +if (ioq !=NULL ) { +/* Drop the reference taken by the io group */ +elv_put_ioq (ioq); + *ioq_ptr =NULL ; + } +} + +/* +* Normally next io queue to be served is selected from the service tree. +* This function allows one to choose a specific io queue to run next +* out of order. This is primarily to accomodate the close_cooperator +* feature of cfq. +* +* Currently it is done only for root level as to begin with supporting +* close cooperator feature only for root group to make sure default +* cfq behavior in flat hierarchy is not changed. +*/ +void elv_set_next_ioq (struct request_queue *q,struct io_queue *ioq) +{ +struct elv_fq_data *efqd = &q->elevator->efqd; +struct io_entity *entity = &ioq->entity; +struct io_sched_data *sd = &efqd->root_group->sched_data; +struct io_service_tree *st =io_entity_service_tree (entity); + +BUG_ON (efqd->active_queue !=NULL || sd->active_entity !=NULL ); +BUG_ON (!efqd->busy_queues); +BUG_ON (sd != entity->sched_data); +BUG_ON (!st); + +bfq_update_vtime (st); +bfq_active_extract (st, entity); + sd->active_entity = entity; + entity->service =0 ; +elv_log_ioq (efqd, ioq,"set_next_ioq" ); +} + +/* Get next queue for service. */ +struct io_queue *elv_get_next_ioq (struct request_queue *q,int extract) +{ +struct elv_fq_data *efqd = &q->elevator->efqd; +struct io_entity *entity =NULL ; +struct io_queue *ioq =NULL ; +struct io_sched_data *sd; + +/* +* We should not call lookup when an entity is active, as doing +* lookup can result in an erroneous vtime jump. +*/ +BUG_ON (efqd->active_queue !=NULL ); + +if (!efqd->busy_queues) +return NULL ; + + sd = &efqd->root_group->sched_data; + entity =bfq_lookup_next_entity (sd,1 ); +
block/elevator-fq.c line 1145
Quoted 10 lines
block/elevator-fq.c new file +10
-0+BUG_ON (!entity); +if (extract) + entity->service =0 ; + ioq =io_entity_to_ioq (entity); + +return ioq; +} + +/* +* coop tells that io scheduler selected a queue for us and we did not
coop?
Quoted 888 lines
block/elevator-fq.c new file +870
-0+* select the next queue based on fairness. +*/ +static void __elv_set_active_ioq (struct elv_fq_data *efqd,struct io_queue *ioq, +int coop) +{ +struct request_queue *q = efqd->queue; + +if (ioq) { +elv_log_ioq (efqd, ioq,"set_active, busy=%d" , + efqd->busy_queues); + ioq->slice_end =0 ; + +elv_clear_ioq_wait_request (ioq); +elv_clear_ioq_must_dispatch (ioq); +elv_mark_ioq_slice_new (ioq); + +del_timer (&efqd->idle_slice_timer); + } + + efqd->active_queue = ioq; + +/* Let iosched know if it wants to take some action */ +if (ioq) { +if (q->elevator->ops->elevator_active_ioq_set_fn) + q->elevator->ops->elevator_active_ioq_set_fn (q, + ioq->sched_queue, coop); + } +} + +/* Get and set a new active queue for service. */ +struct io_queue *elv_set_active_ioq (struct request_queue *q, +struct io_queue *ioq) +{ +struct elv_fq_data *efqd = &q->elevator->efqd; +int coop =0 ; + +if (!ioq) + ioq =elv_get_next_ioq (q,1 ); +else { +elv_set_next_ioq (q, ioq); +/* +* io scheduler selected the next queue for us. Pass this +* this info back to io scheudler. cfq currently uses it +* to reset coop flag on the queue. +*/ + coop =1 ; + } +__elv_set_active_ioq (efqd, ioq, coop); +return ioq; +} + +void elv_reset_active_ioq (struct elv_fq_data *efqd) +{ +struct request_queue *q = efqd->queue; +struct io_queue *ioq =elv_active_ioq (efqd->queue->elevator); + +if (q->elevator->ops->elevator_active_ioq_reset_fn) + q->elevator->ops->elevator_active_ioq_reset_fn (q, + ioq->sched_queue); + efqd->active_queue =NULL ; +del_timer (&efqd->idle_slice_timer); +} + +void elv_activate_ioq (struct io_queue *ioq,int add_front) +{ +bfq_activate_entity (&ioq->entity, add_front); +} + +void elv_deactivate_ioq (struct elv_fq_data *efqd,struct io_queue *ioq, +int requeue) +{ +bfq_deactivate_entity (&ioq->entity, requeue); +} + +/* Called when an inactive queue receives a new request. */ +void elv_add_ioq_busy (struct elv_fq_data *efqd,struct io_queue *ioq) +{ +BUG_ON (elv_ioq_busy (ioq)); +BUG_ON (ioq == efqd->active_queue); +elv_log_ioq (efqd, ioq,"add to busy" ); +elv_activate_ioq (ioq,0 ); +elv_mark_ioq_busy (ioq); + efqd->busy_queues++; +if (elv_ioq_class_rt (ioq)) { +struct io_group *iog =ioq_to_io_group (ioq); + iog->busy_rt_queues++; + } +} + +void elv_del_ioq_busy (struct elevator_queue *e,struct io_queue *ioq, +int requeue) +{ +struct elv_fq_data *efqd = &e->efqd; + +BUG_ON (!elv_ioq_busy (ioq)); +BUG_ON (ioq->nr_queued); +elv_log_ioq (efqd, ioq,"del from busy" ); +elv_clear_ioq_busy (ioq); +BUG_ON (efqd->busy_queues ==0 ); + efqd->busy_queues--; +if (elv_ioq_class_rt (ioq)) { +struct io_group *iog =ioq_to_io_group (ioq); + iog->busy_rt_queues--; + } + +elv_deactivate_ioq (efqd, ioq, requeue); +} + +/* +* Do the accounting. Determine how much service (in terms of time slices) +* current queue used and adjust the start, finish time of queue and vtime +* of the tree accordingly. +* +* Determining the service used in terms of time is tricky in certain +* situations. Especially when underlying device supports command queuing +* and requests from multiple queues can be there at same time, then it +* is not clear which queue consumed how much of disk time. +* +* To mitigate this problem, cfq starts the time slice of the queue only +* after first request from the queue has completed. This does not work +* very well if we expire the queue before we wait for first and more +* request to finish from the queue. For seeky queues, we will expire the +* queue after dispatching few requests without waiting and start dispatching +* from next queue. +* +* Not sure how to determine the time consumed by queue in such scenarios. +* Currently as a crude approximation, we are charging 25% of time slice +* for such cases. A better mechanism is needed for accurate accounting. +*/ +void __elv_ioq_slice_expired (struct request_queue *q,struct io_queue *ioq) +{ +struct elv_fq_data *efqd = &q->elevator->efqd; +struct io_entity *entity = &ioq->entity; +long slice_unused =0 , slice_used =0 , slice_overshoot =0 ; + +assert_spin_locked (q->queue_lock); +elv_log_ioq (efqd, ioq,"slice expired" ); + +if (elv_ioq_wait_request (ioq)) +del_timer (&efqd->idle_slice_timer); + +elv_clear_ioq_wait_request (ioq); + +/* +* if ioq->slice_end = 0, that means a queue was expired before first +* reuqest from the queue got completed. Of course we are not planning +* to idle on the queue otherwise we would not have expired it. +* +* Charge for the 25% slice in such cases. This is not the best thing +* to do but at the same time not very sure what's the next best +* thing to do. +* +* This arises from that fact that we don't have the notion of +* one queue being operational at one time. io scheduler can dispatch +* requests from multiple queues in one dispatch round. Ideally for +* more accurate accounting of exact disk time used by disk, one +* should dispatch requests from only one queue and wait for all +* the requests to finish. But this will reduce throughput. +*/ +if (!ioq->slice_end) + slice_used = entity->budget/4 ; +else { +if (time_after (ioq->slice_end, jiffies)) { + slice_unused = ioq->slice_end - jiffies; +if (slice_unused == entity->budget) { +/* +* queue got expired immediately after +* completing first request. Charge 25% of +* slice. +*/ + slice_used = entity->budget/4 ; + }else + slice_used = entity->budget - slice_unused; + }else { + slice_overshoot = jiffies - ioq->slice_end; + slice_used = entity->budget + slice_overshoot; + } + } + +elv_log_ioq (efqd, ioq,"sl_end=%lx, jiffies=%lx" , ioq->slice_end, + jiffies); +elv_log_ioq (efqd, ioq,"sl_used=%ld, budget=%ld overshoot=%ld" , + slice_used, entity->budget, slice_overshoot); +elv_ioq_served (ioq, slice_used); + +BUG_ON (ioq != efqd->active_queue); +elv_reset_active_ioq (efqd); + +if (!ioq->nr_queued) +elv_del_ioq_busy (q->elevator, ioq,1 ); +else +elv_activate_ioq (ioq,0 ); +} +EXPORT_SYMBOL (__elv_ioq_slice_expired); + +/* +* Expire the ioq. +*/ +void elv_ioq_slice_expired (struct request_queue *q) +{ +struct io_queue *ioq =elv_active_ioq (q->elevator); + +if (ioq) +__elv_ioq_slice_expired (q, ioq); +} + +/* +* Check if new_cfqq should preempt the currently active queue. Return 0 for +* no or if we aren't sure, a 1 will cause a preemption attempt. +*/ +int elv_should_preempt (struct request_queue *q,struct io_queue *new_ioq, +struct request *rq) +{ +struct io_queue *ioq; +struct elevator_queue *eq = q->elevator; +struct io_entity *entity, *new_entity; + + ioq =elv_active_ioq (eq); + +if (!ioq) +return 0 ; + + entity = &ioq->entity; + new_entity = &new_ioq->entity; + +/* +* Allow an RT request to pre-empt an ongoing non-RT cfqq timeslice. +*/ + +if (new_entity->ioprio_class ==IOPRIO_CLASS_RT + && entity->ioprio_class !=IOPRIO_CLASS_RT ) +return 1 ; +/* +* Allow an BE request to pre-empt an ongoing IDLE clas timeslice. +*/ + +if (new_entity->ioprio_class ==IOPRIO_CLASS_BE + && entity->ioprio_class ==IOPRIO_CLASS_IDLE ) +return 1 ; + +/* +* Check with io scheduler if it has additional criterion based on +* which it wants to preempt existing queue. +*/ +if (eq->ops->elevator_should_preempt_fn) +return eq->ops->elevator_should_preempt_fn (q, +ioq_sched_queue (new_ioq), rq); + +return 0 ; +} + +static void elv_preempt_queue (struct request_queue *q,struct io_queue *ioq) +{ +elv_log_ioq (&q->elevator->efqd, ioq,"preempt" ); +elv_ioq_slice_expired (q); + +/* +* Put the new queue at the front of the of the current list, +* so we know that it will be selected next. +*/ + +elv_activate_ioq (ioq,1 ); +elv_ioq_set_slice_end (ioq,0 ); +elv_mark_ioq_slice_new (ioq); +} + +void elv_ioq_request_add (struct request_queue *q,struct request *rq) +{ +struct elv_fq_data *efqd = &q->elevator->efqd; +struct io_queue *ioq = rq->ioq; + +if (!elv_iosched_fair_queuing_enabled (q->elevator)) +return ; + +BUG_ON (!efqd); +BUG_ON (!ioq); + efqd->rq_queued++; + ioq->nr_queued++; + +if (!elv_ioq_busy (ioq)) +elv_add_ioq_busy (efqd, ioq); + +elv_ioq_update_io_thinktime (ioq); +elv_ioq_update_idle_window (q->elevator, ioq, rq); + +if (ioq ==elv_active_ioq (q->elevator)) { +/* +* Remember that we saw a request from this process, but +* don't start queuing just yet. Otherwise we risk seeing lots +* of tiny requests, because we disrupt the normal plugging +* and merging. If the request is already larger than a single +* page, let it rip immediately. For that case we assume that +* merging is already done. Ditto for a busy system that +* has other work pending, don't risk delaying until the +* idle timer unplug to continue working. +*/ +if (elv_ioq_wait_request (ioq)) { +if (blk_rq_bytes (rq) >PAGE_CACHE_SIZE || + efqd->busy_queues >1 ) { +del_timer (&efqd->idle_slice_timer); +blk_start_queueing (q); + } +elv_mark_ioq_must_dispatch (ioq); + } + }else if (elv_should_preempt (q, ioq, rq)) { +/* +* not the active queue - expire current slice if it is +* idle and has expired it's mean thinktime or this new queue +* has some old slice time left and is of higher priority or +* this new queue is RT and the current one is BE +*/ +elv_preempt_queue (q, ioq); +blk_start_queueing (q); + } +} + +void elv_idle_slice_timer (unsigned long data) +{ +struct elv_fq_data *efqd = (struct elv_fq_data *)data; +struct io_queue *ioq; +unsigned long flags; +struct request_queue *q = efqd->queue; + +elv_log (efqd,"idle timer fired" ); + +spin_lock_irqsave (q->queue_lock, flags); + + ioq = efqd->active_queue; + +if (ioq) { + +/* +* We saw a request before the queue expired, let it through +*/ +if (elv_ioq_must_dispatch (ioq)) + goto out_kick; + +/* +* expired +*/ +if (elv_ioq_slice_used (ioq)) + goto expire; + +/* +* only expire and reinvoke request handler, if there are +* other queues with pending requests +*/ +if (!elv_nr_busy_ioq (q->elevator)) + goto out_cont; + +/* +* not expired and it has a request pending, let it dispatch +*/ +if (ioq->nr_queued) + goto out_kick; + } +expire: +elv_ioq_slice_expired (q); +out_kick: +elv_schedule_dispatch (q); +out_cont: +spin_unlock_irqrestore (q->queue_lock, flags); +} + +void elv_ioq_arm_slice_timer (struct request_queue *q) +{ +struct elv_fq_data *efqd = &q->elevator->efqd; +struct io_queue *ioq =elv_active_ioq (q->elevator); +unsigned long sl; + +BUG_ON (!ioq); + +/* +* SSD device without seek penalty, disable idling. But only do so +* for devices that support queuing, otherwise we still have a problem +* with sync vs async workloads. +*/ +if (blk_queue_nonrot (q) && efqd->hw_tag) +return ; + +/* +* still requests with the driver, don't idle +*/ +if (efqd->rq_in_driver) +return ; + +/* +* idle is disabled, either manually or by past process history +*/ +if (!efqd->elv_slice_idle || !elv_ioq_idle_window (ioq)) +return ; + +/* +* may be iosched got its own idling logic. In that case io +* schduler will take care of arming the timer, if need be. +*/ +if (q->elevator->ops->elevator_arm_slice_timer_fn) { + q->elevator->ops->elevator_arm_slice_timer_fn (q, + ioq->sched_queue); + }else { +elv_mark_ioq_wait_request (ioq); + sl = efqd->elv_slice_idle; +mod_timer (&efqd->idle_slice_timer, jiffies + sl); +elv_log_ioq (efqd, ioq,"arm idle: %lu" , sl); + } +} + +/* Common layer function to select the next queue to dispatch from */ +void *elv_fq_select_ioq (struct request_queue *q,int force) +{ +struct elv_fq_data *efqd = &q->elevator->efqd; +struct io_queue *new_ioq =NULL , *ioq =elv_active_ioq (q->elevator); +struct io_group *iog; + +if (!elv_nr_busy_ioq (q->elevator)) +return NULL ; + +if (ioq ==NULL ) + goto new_queue; + +/* +* Force dispatch. Continue to dispatch from current queue as long +* as it has requests. +*/ +if (unlikely (force)) { +if (ioq->nr_queued) + goto keep_queue; +else + goto expire; + } + +/* +* The active queue has run out of time, expire it and select new. +*/ +if (elv_ioq_slice_used (ioq) && !elv_ioq_must_dispatch (ioq)) + goto expire; + +/* +* If we have a RT cfqq waiting, then we pre-empt the current non-rt +* cfqq. +*/ + iog =ioq_to_io_group (ioq); + +if (!elv_ioq_class_rt (ioq) && iog->busy_rt_queues) { +/* +* We simulate this as cfqq timed out so that it gets to bank +* the remaining of its time slice. +*/ +elv_log_ioq (efqd, ioq,"preempt" ); + goto expire; + } + +/* +* The active queue has requests and isn't expired, allow it to +* dispatch. +*/ + +if (ioq->nr_queued) + goto keep_queue; + +/* +* If another queue has a request waiting within our mean seek +* distance, let it run. The expire code will check for close +* cooperators and put the close queue at the front of the service +* tree. +*/ + new_ioq =elv_close_cooperator (q, ioq,0 ); +if (new_ioq) + goto expire; + +/* +* No requests pending. If the active queue still has requests in +* flight or is idling for a new request, allow either of these +* conditions to happen (or time out) before selecting a new queue. +*/ + +if (timer_pending (&efqd->idle_slice_timer) || + (elv_ioq_nr_dispatched (ioq) &&elv_ioq_idle_window (ioq))) { + ioq =NULL ; + goto keep_queue; + } + +expire: +elv_ioq_slice_expired (q); +new_queue: + ioq =elv_set_active_ioq (q, new_ioq); +keep_queue: +return ioq; +} + +/* A request got removed from io_queue. Do the accounting */ +void elv_ioq_request_removed (struct elevator_queue *e,struct request *rq) +{ +struct io_queue *ioq; +struct elv_fq_data *efqd; + +if (!elv_iosched_fair_queuing_enabled (e)) +return ; + + ioq = rq->ioq; +BUG_ON (!ioq); + ioq->nr_queued--; + + efqd = ioq->efqd; +BUG_ON (!efqd); + efqd->rq_queued--; + +if (elv_ioq_busy (ioq) && (elv_active_ioq (e) != ioq) && !ioq->nr_queued) +elv_del_ioq_busy (e, ioq,1 ); +} + +/* A request got dispatched. Do the accounting. */ +void elv_fq_dispatched_request (struct elevator_queue *e,struct request *rq) +{ +struct io_queue *ioq = rq->ioq; + +if (!elv_iosched_fair_queuing_enabled (e)) +return ; + +BUG_ON (!ioq); +elv_ioq_request_dispatched (ioq); +elv_ioq_request_removed (e, rq); +elv_clear_ioq_must_dispatch (ioq); +} + +void elv_fq_activate_rq (struct request_queue *q,struct request *rq) +{ +struct elv_fq_data *efqd = &q->elevator->efqd; + +if (!elv_iosched_fair_queuing_enabled (q->elevator)) +return ; + + efqd->rq_in_driver++; +elv_log_ioq (efqd,rq_ioq (rq),"activate rq, drv=%d" , + efqd->rq_in_driver); +} + +void elv_fq_deactivate_rq (struct request_queue *q,struct request *rq) +{ +struct elv_fq_data *efqd = &q->elevator->efqd; + +if (!elv_iosched_fair_queuing_enabled (q->elevator)) +return ; + +WARN_ON (!efqd->rq_in_driver); + efqd->rq_in_driver--; +elv_log_ioq (efqd,rq_ioq (rq),"deactivate rq, drv=%d" , + efqd->rq_in_driver); +} + +/* +* Update hw_tag based on peak queue depth over 50 samples under +* sufficient load. +*/ +static void elv_update_hw_tag (struct elv_fq_data *efqd) +{ +if (efqd->rq_in_driver > efqd->rq_in_driver_peak) + efqd->rq_in_driver_peak = efqd->rq_in_driver; + +if (efqd->rq_queued <=ELV_HW_QUEUE_MIN && + efqd->rq_in_driver <=ELV_HW_QUEUE_MIN ) +return ; + +if (efqd->hw_tag_samples++ <50 ) +return ; + +if (efqd->rq_in_driver_peak >=ELV_HW_QUEUE_MIN ) + efqd->hw_tag =1 ; +else + efqd->hw_tag =0 ; + + efqd->hw_tag_samples =0 ; + efqd->rq_in_driver_peak =0 ; +} + +/* +* If ioscheduler has functionality of keeping track of close cooperator, check +* with it if it has got a closely co-operating queue. +*/ +static inline struct io_queue *elv_close_cooperator (struct request_queue *q, +struct io_queue *ioq,int probe) +{ +struct elevator_queue *e = q->elevator; +struct io_queue *new_ioq =NULL ; + +/* +* Currently this feature is supported only for flat hierarchy or +* root group queues so that default cfq behavior is not changed. +*/ +if (!is_root_group_ioq (q, ioq)) +return NULL ; + +if (q->elevator->ops->elevator_close_cooperator_fn) + new_ioq = e->ops->elevator_close_cooperator_fn (q, + ioq->sched_queue, probe); + +/* Only select co-operating queue if it belongs to root group */ +if (new_ioq && !is_root_group_ioq (q, new_ioq)) +return NULL ; + +return new_ioq; +} + +/* A request got completed from io_queue. Do the accounting. */ +void elv_ioq_completed_request (struct request_queue *q,struct request *rq) +{ +const int sync =rq_is_sync (rq); +struct io_queue *ioq; +struct elv_fq_data *efqd = &q->elevator->efqd; + +if (!elv_iosched_fair_queuing_enabled (q->elevator)) +return ; + + ioq = rq->ioq; + +elv_log_ioq (efqd, ioq,"complete" ); + +elv_update_hw_tag (efqd); + +WARN_ON (!efqd->rq_in_driver); +WARN_ON (!ioq->dispatched); + efqd->rq_in_driver--; + ioq->dispatched--; + +if (sync) + ioq->last_end_request = jiffies; + +/* +* If this is the active queue, check if it needs to be expired, +* or if we want to idle in case it has no pending requests. +*/ + +if (elv_active_ioq (q->elevator) == ioq) { +if (elv_ioq_slice_new (ioq)) { +elv_ioq_set_prio_slice (q, ioq); +elv_clear_ioq_slice_new (ioq); + } +/* +* If there are no requests waiting in this queue, and +* there are other queues ready to issue requests, AND +* those other queues are issuing requests within our +* mean seek distance, give them a chance to run instead +* of idling. +*/ +if (elv_ioq_slice_used (ioq) ||elv_ioq_class_idle (ioq)) +elv_ioq_slice_expired (q); +else if (!ioq->nr_queued && !elv_close_cooperator (q, ioq,1 ) + && sync && !rq_noidle (rq)) +elv_ioq_arm_slice_timer (q); + } + +if (!efqd->rq_in_driver) +elv_schedule_dispatch (q); +} + +struct io_group *io_lookup_io_group_current (struct request_queue *q) +{ +struct elv_fq_data *efqd = &q->elevator->efqd; + +return efqd->root_group; +} +EXPORT_SYMBOL (io_lookup_io_group_current); + +void *io_group_async_queue_prio (struct io_group *iog,int ioprio_class, +int ioprio) +{ +struct io_queue *ioq =NULL ; + +switch (ioprio_class) { +case IOPRIO_CLASS_RT : + ioq = iog->async_queue[0 ][ioprio]; +break ; +case IOPRIO_CLASS_BE : + ioq = iog->async_queue[1 ][ioprio]; +break ; +case IOPRIO_CLASS_IDLE : + ioq = iog->async_idle_queue; +break ; +default : +BUG (); + } + +if (ioq) +return ioq->sched_queue; +return NULL ; +} +EXPORT_SYMBOL (io_group_async_queue_prio); + +void io_group_set_async_queue (struct io_group *iog,int ioprio_class, +int ioprio,struct io_queue *ioq) +{ +switch (ioprio_class) { +case IOPRIO_CLASS_RT : + iog->async_queue[0 ][ioprio] = ioq; +break ; +case IOPRIO_CLASS_BE : + iog->async_queue[1 ][ioprio] = ioq; +break ; +case IOPRIO_CLASS_IDLE : + iog->async_idle_queue = ioq; +break ; +default : +BUG (); + } + +/* +* Take the group reference and pin the queue. Group exit will +* clean it up +*/ +elv_get_ioq (ioq); +} +EXPORT_SYMBOL (io_group_set_async_queue); + +/* +* Release all the io group references to its async queues. +*/ +void io_put_io_group_queues (struct elevator_queue *e,struct io_group *iog) +{ +int i, j; + +for (i =0 ; i <2 ; i++) +for (j =0 ; j <IOPRIO_BE_NR ; j++) +elv_release_ioq (e, &iog->async_queue[i][j]); + +/* Free up async idle queue */ +elv_release_ioq (e, &iog->async_idle_queue); +} + +struct io_group *io_alloc_root_group (struct request_queue *q, +struct elevator_queue *e,void *key) +{ +struct io_group *iog; +int i; + + iog =kmalloc_node (sizeof (*iog),GFP_KERNEL | __GFP_ZERO, q->node); +if (iog ==NULL ) +return NULL ; + +for (i =0 ; i <IO_IOPRIO_CLASSES ; i++) + iog->sched_data.service_tree[i] =IO_SERVICE_TREE_INIT ; + +return iog; +} + +void io_free_root_group (struct elevator_queue *e) +{ +struct io_group *iog = e->efqd.root_group; +struct io_service_tree *st; +int i; + +for (i =0 ; i <IO_IOPRIO_CLASSES ; i++) { + st = iog->sched_data.service_tree + i; +io_flush_idle_tree (st); + } + +io_put_io_group_queues (e, iog); +kfree (iog); +} + +static void elv_slab_kill (void ) +{ +/* +* Caller already ensured that pending RCU callbacks are completed, +* so we should have no busy allocations at this point. +*/ +if (elv_ioq_pool) +kmem_cache_destroy (elv_ioq_pool); +} + +static int __init elv_slab_setup (void ) +{ + elv_ioq_pool =KMEM_CACHE (io_queue,0 ); +if (!elv_ioq_pool) + goto fail; + +return 0 ; +fail: +elv_slab_kill (); +return -ENOMEM ; +} + +/* Initialize fair queueing data associated with elevator */ +int elv_init_fq_data (struct request_queue *q,struct elevator_queue *e) +{ +struct io_group *iog; +struct elv_fq_data *efqd = &e->efqd; + +if (!elv_iosched_fair_queuing_enabled (e)) +return 0 ; + + iog =io_alloc_root_group (q, e, efqd); +if (iog ==NULL ) +return 1 ; + + efqd->root_group = iog; + efqd->queue = q; + +init_timer (&efqd->idle_slice_timer); + efqd->idle_slice_timer.function = elv_idle_slice_timer; + efqd->idle_slice_timer.data = (unsigned long ) efqd; + +INIT_WORK (&efqd->unplug_work, elv_kick_queue); + + efqd->elv_slice[0 ] = elv_slice_async; + efqd->elv_slice[1 ] = elv_slice_sync; + efqd->elv_slice_idle = elv_slice_idle; + efqd->hw_tag =1 ; + +return 0 ; +} + +/* +* elv_exit_fq_data is called before we call elevator_exit_fn. Before +* we ask elevator to cleanup its queues, we do the cleanup here so +* that all the group and idle tree references to ioq are dropped. Later +* during elevator cleanup, ioc reference will be dropped which will lead +* to removal of ioscheduler queue as well as associated ioq object. +*/ +void elv_exit_fq_data (struct elevator_queue *e) +{ +struct elv_fq_data *efqd = &e->efqd; + +if (!elv_iosched_fair_queuing_enabled (e)) +return ; + +elv_shutdown_timer_wq (e); + +BUG_ON (timer_pending (&efqd->idle_slice_timer)); +io_free_root_group (e); +} + +/* +* This is called after the io scheduler has cleaned up its data structres. +* I don't think that this function is required. Right now just keeping it +* because cfq cleans up timer and work queue again after freeing up +* io contexts. To me io scheduler has already been drained out, and all +* the active queue have already been expired so time and work queue should +* not been activated during cleanup process. +* +* Keeping it here for the time being. Will get rid of it later. +*/ +void elv_exit_fq_data_post (struct elevator_queue *e) +{ +struct elv_fq_data *efqd = &e->efqd; + +if (!elv_iosched_fair_queuing_enabled (e)) +return ; + +elv_shutdown_timer_wq (e); +BUG_ON (timer_pending (&efqd->idle_slice_timer)); +} + + +static int __init elv_fq_init (void ) +{ +if (elv_slab_setup ()) +return -ENOMEM ; + +/* could be 0 on HZ < 1000 setups */ + +if (!elv_slice_async) + elv_slice_async =1 ; + +if (!elv_slice_idle) + elv_slice_idle =1 ; + +return 0 ; +} + +module_init (elv_fq_init);block/elevator-fq.h new file +12
-0@@ -0,0 +1,473 @@ +/* +* BFQ: data structures and common functions prototypes. +* +* Based on ideas and code from CFQ: +* Copyright (C) 2003 Jens Axboe <[email protected]> +* +* Copyright (C) 2008 Fabio Checconi <[email protected]> +* Paolo Valente <[email protected]> +* Copyright (C) 2009 Vivek Goyal <[email protected]> +* Nauman Rafique <[email protected]> +*/ +
block/elevator-fq.h line 22
Quoted 10 lines
block/elevator-fq.h new file +10
-0+#include <linux/blkdev.h> + +#ifndef _BFQ_SCHED_H +#define _BFQ_SCHED_H + +#define IO_IOPRIO_CLASSES 3 + +typedef u64 bfq_timestamp_t ; +typedef unsigned long bfq_weight_t ; +typedef unsigned long bfq_service_t ;
Does this abstraction really provide any benefit? Why not directly use the standard C types, make the code easier to read.
block/elevator-fq.h line 32
Quoted 10 lines
block/elevator-fq.h new file +10
-0+struct io_entity ; +struct io_queue ; + +#ifdef CONFIG_ELV_FAIR_QUEUING + +#define ELV_ATTR (name) \ + __ATTR(name, S_IRUGO|S_IWUSR, elv_##name##_show, elv_##name##_store) + +/** +* struct bfq_service_tree - per ioprio_class service tree.
Comment is old, does not reflect the newer name
Quoted 15 lines
block/elevator-fq.h new file +15
-0+* @active: tree for active entities (i.e., those backlogged). +* @idle: tree for idle entities (i.e., those not backlogged, with V <= F_i). +* @first_idle: idle entity with minimum F_i. +* @last_idle: idle entity with maximum F_i. +* @vtime: scheduler virtual time. +* @wsum: scheduler weight sum; active and idle entities contribute to it. +* +* Each service tree represents a B-WF2Q+ scheduler on its own. Each +* ioprio_class has its own independent scheduler, and so its own +* bfq_service_tree. All the fields are protected by the queue lock +* of the containing efqd. +*/ +struct io_service_tree { +struct rb_root active; +struct rb_root idle;
block/elevator-fq.h line 57
Quoted 10 lines
block/elevator-fq.h new file +10
-0+ +struct io_entity *first_idle; +struct io_entity *last_idle; + +bfq_timestamp_t vtime; +bfq_weight_t wsum; +}; + +/** +* struct bfq_sched_data - multi-class scheduler.
Again the naming convention is broken, you need to change several bfq's to io's :)
Quoted 19 lines
block/elevator-fq.h new file +19
-0+* @active_entity: entity under service. +* @next_active: head-of-the-line entity in the scheduler. +* @service_tree: array of service trees, one per ioprio_class. +* +* bfq_sched_data is the basic scheduler queue. It supports three +* ioprio_classes, and can be used either as a toplevel queue or as +* an intermediate queue on a hierarchical setup. +* @next_active points to the active entity of the sched_data service +* trees that will be scheduled next. +* +* The supported ioprio_classes are the same as in CFQ, in descending +* priority order, IOPRIO_CLASS_RT, IOPRIO_CLASS_BE, IOPRIO_CLASS_IDLE. +* Requests from higher priority queues are served before all the +* requests from lower priority queues; among requests of the same +* queue requests are served according to B-WF2Q+. +* All the fields are protected by the queue lock of the containing bfqd. +*/ +struct io_sched_data { +struct io_entity *active_entity;
block/elevator-fq.h line 86
Quoted 10 lines
block/elevator-fq.h new file +10
-0+struct io_service_tree service_tree[IO_IOPRIO_CLASSES ]; +}; + +/** +* struct bfq_entity - schedulable entity. +* @rb_node: service_tree member. +* @on_st: flag, true if the entity is on a tree (either the active or +* the idle one of its service_tree). +* @finish: B-WF2Q+ finish timestamp (aka F_i). +* @start: B-WF2Q+ start timestamp (aka S_i).
Could you mention what key is used in the rb_tree? start, finish sounds like a range, so my suspicion is that start is used.
block/elevator-fq.h line 89
Quoted 3 lines
block/elevator-fq.h new file +3
-0+* @tree: tree the entity is enqueued into; %NULL if not on a tree. +* @min_start: minimum start time of the (active) subtree rooted at +* this entity; used for O(log N) lookups into active trees.
Used for O(log N) makes no sense to me, RBTree has a worst case lookup time of O(log N), but what is the comment saying?
Quoted 59 lines
block/elevator-fq.h new file +59
-0+* @service: service received during the last round of service. +* @budget: budget used to calculate F_i; F_i = S_i + @budget / @weight. +* @weight: weight of the queue, calculated as IOPRIO_BE_NR - @ioprio. +* @parent: parent entity, for hierarchical scheduling. +* @my_sched_data: for non-leaf nodes in the cgroup hierarchy, the +* associated scheduler queue, %NULL on leaf nodes. +* @sched_data: the scheduler queue this entity belongs to. +* @ioprio: the ioprio in use. +* @new_ioprio: when an ioprio change is requested, the new ioprio value +* @ioprio_class: the ioprio_class in use. +* @new_ioprio_class: when an ioprio_class change is requested, the new +* ioprio_class value. +* @ioprio_changed: flag, true when the user requested an ioprio or +* ioprio_class change. +* +* A bfq_entity is used to represent either a bfq_queue (leaf node in the +* cgroup hierarchy) or a bfq_group into the upper level scheduler. Each +* entity belongs to the sched_data of the parent group in the cgroup +* hierarchy. Non-leaf entities have also their own sched_data, stored +* in @my_sched_data. +* +* Each entity stores independently its priority values; this would allow +* different weights on different devices, but this functionality is not +* exported to userspace by now. Priorities are updated lazily, first +* storing the new values into the new_* fields, then setting the +* @ioprio_changed flag. As soon as there is a transition in the entity +* state that allows the priority update to take place the effective and +* the requested priority values are synchronized. +* +* The weight value is calculated from the ioprio to export the same +* interface as CFQ. When dealing with ``well-behaved'' queues (i.e., +* queues that do not spend too much time to consume their budget and +* have true sequential behavior, and when there are no external factors +* breaking anticipation) the relative weights at each level of the +* cgroups hierarchy should be guaranteed. +* All the fields are protected by the queue lock of the containing bfqd. +*/ +struct io_entity { +struct rb_node rb_node; + +int on_st; + +bfq_timestamp_t finish; +bfq_timestamp_t start; + +struct rb_root *tree; + +bfq_timestamp_t min_start; + +bfq_service_t service, budget; +bfq_weight_t weight; + +struct io_entity *parent; + +struct io_sched_data *my_sched_data; +struct io_sched_data *sched_data; + +unsigned short ioprio, new_ioprio; +unsigned short ioprio_class, new_ioprio_class;
block/elevator-fq.h line 158
Quoted 10 lines
block/elevator-fq.h new file +10
-0+ +int ioprio_changed; +}; + +/* +* A common structure embedded by every io scheduler into their respective +* queue structure. +*/ +struct io_queue { +struct io_entity entity;
So the io_queue has an abstract entity called io_entity that contains it QoS parameters? Correct?
block/elevator-fq.h line 164
Quoted 6 lines
block/elevator-fq.h new file +6
-0+atomic_t ref; +unsigned int flags; + +/* Pointer to generic elevator data structure */ +struct elv_fq_data *efqd; +pid_t pid;
Why do we store the pid?
Quoted 14 lines
block/elevator-fq.h new file +14
-0+ +/* Number of requests queued on this io queue */ +unsigned long nr_queued; + +/* Requests dispatched from this queue */ +int dispatched; + +/* Keep a track of think time of processes in this queue */ +unsigned long last_end_request; +unsigned long ttime_total; +unsigned long ttime_samples; +unsigned long ttime_mean; + +unsigned long slice_end;
block/elevator-fq.h line 188
Quoted 10 lines
block/elevator-fq.h new file +10
-0+ +/* Pointer to io scheduler's queue */ +void *sched_queue; +}; + +struct io_group { +struct io_sched_data sched_data; + +/* async_queue and idle_queue are used only for cfq */ +struct io_queue *async_queue[2 ][IOPRIO_BE_NR ];
Again the 2 is confusing
block/elevator-fq.h line 198
Quoted 10 lines
block/elevator-fq.h new file +10
-0+struct io_queue *async_idle_queue; + +/* +* Used to track any pending rt requests so we can pre-empt current +* non-RT cfqq in service when this value is non-zero. +*/ +unsigned int busy_rt_queues; +}; + +struct elv_fq_data {
What does fq stand for?
Quoted 5 lines
block/elevator-fq.h new file +5
-0+struct io_group *root_group; + +struct request_queue *queue; +unsigned int busy_queues; +
block/elevator-fq.h line 213
Quoted 10 lines
block/elevator-fq.h new file +10
-0+/* Number of requests queued */ +int rq_queued; + +/* Pointer to the ioscheduler queue being served */ +void *active_queue; + +int rq_in_driver; +int hw_tag; +int hw_tag_samples; +int rq_in_driver_peak;
Some comments of _in_driver and _in_driver_peak would be nice.
Quoted 4 lines
block/elevator-fq.h new file +4
-0+ +/* +* elevator fair queuing layer has the capability to provide idling +* for ensuring fairness for processes doing dependent reads.
block/elevator-fq.h line 227
Quoted 10 lines
block/elevator-fq.h new file +10
-0+* This might be needed to ensure fairness among two processes doing +* synchronous reads in two different cgroups. noop and deadline don't +* have any notion of anticipation/idling. As of now, these are the +* users of this functionality. +*/ +unsigned int elv_slice_idle; +struct timer_list idle_slice_timer; +struct work_struct unplug_work; + +unsigned int elv_slice[2 ];
Why [2] makes the code hearder to read
Quoted 492 lines
block/elevator-fq.h new file +246
-0+}; + +extern int elv_slice_idle; +extern int elv_slice_async; + +/* Logging facilities. */ +#define elv_log_ioq(efqd, ioq, fmt, args...) \ + blk_add_trace_msg((efqd)->queue, "elv%d%c " fmt, (ioq)->pid, \ + elv_ioq_sync(ioq) ? 'S' : 'A', ##args) + +#define elv_log(efqd, fmt, args...) \ + blk_add_trace_msg((efqd)->queue, "elv " fmt, ##args) + +#define ioq_sample_valid (samples) ((samples) > 80) + +/* Some shared queue flag manipulation functions among elevators */ + +enum elv_queue_state_flags { + ELV_QUEUE_FLAG_busy =0 ,/* has requests or is under service */ + ELV_QUEUE_FLAG_sync,/* synchronous queue */ + ELV_QUEUE_FLAG_idle_window,/* elevator slice idling enabled */ + ELV_QUEUE_FLAG_wait_request,/* waiting for a request */ + ELV_QUEUE_FLAG_must_dispatch,/* must be allowed a dispatch */ + ELV_QUEUE_FLAG_slice_new,/* no requests dispatched in slice */ +ELV_QUEUE_FLAG_NR , +}; + +#define ELV_IO_QUEUE_FLAG_FNS (name) \ +static inline void elv_mark_ioq_##name(struct io_queue *ioq) \ +{ \ + (ioq)->flags |= (1 << ELV_QUEUE_FLAG_##name); \ +} \ +static inline void elv_clear_ioq_##name(struct io_queue *ioq) \ +{ \ + (ioq)->flags &= ~(1 << ELV_QUEUE_FLAG_##name); \ +} \ +static inline int elv_ioq_##name(struct io_queue *ioq) \ +{ \ + return ((ioq)->flags & (1 << ELV_QUEUE_FLAG_##name)) != 0; \ +} + +ELV_IO_QUEUE_FLAG_FNS (busy) +ELV_IO_QUEUE_FLAG_FNS (sync) +ELV_IO_QUEUE_FLAG_FNS (wait_request) +ELV_IO_QUEUE_FLAG_FNS(must_dispatch) +ELV_IO_QUEUE_FLAG_FNS(idle_window) +ELV_IO_QUEUE_FLAG_FNS(slice_new) + +static inline struct io_service_tree * +io_entity_service_tree(struct io_entity *entity) +{ + struct io_sched_data *sched_data = entity->sched_data; +unsigned int idx = entity->ioprio_class -1 ; + + BUG_ON(idx >= IO_IOPRIO_CLASSES); + BUG_ON(sched_data == NULL); + + return sched_data->service_tree + idx; +} + +/* A request got dispatched from the io_queue. Do the accounting. */ +static inline void elv_ioq_request_dispatched(struct io_queue *ioq) +{ + ioq->dispatched++; +} + +static inline int elv_ioq_slice_used (struct io_queue *ioq) +{ +if (elv_ioq_slice_new (ioq)) +return 0 ; +if (time_before (jiffies, ioq->slice_end)) +return 0 ; + +return 1 ; +} + +/* How many request are currently dispatched from the queue */ +static inline int elv_ioq_nr_dispatched (struct io_queue *ioq) +{ +return ioq->dispatched; +} + +/* How many request are currently queued in the queue */ +static inline int elv_ioq_nr_queued (struct io_queue *ioq) +{ +return ioq->nr_queued; +} + +static inline void elv_get_ioq (struct io_queue *ioq) +{ +atomic_inc (&ioq->ref); +} + +static inline void elv_ioq_set_slice_end (struct io_queue *ioq, +unsigned long slice_end) +{ + ioq->slice_end = slice_end; +} + +static inline int elv_ioq_class_idle (struct io_queue *ioq) +{ +return ioq->entity.ioprio_class ==IOPRIO_CLASS_IDLE ; +} + +static inline int elv_ioq_class_rt (struct io_queue *ioq) +{ +return ioq->entity.ioprio_class ==IOPRIO_CLASS_RT ; +} + +static inline int elv_ioq_ioprio_class (struct io_queue *ioq) +{ +return ioq->entity.new_ioprio_class; +} + +static inline int elv_ioq_ioprio (struct io_queue *ioq) +{ +return ioq->entity.new_ioprio; +} + +static inline void elv_ioq_set_ioprio_class (struct io_queue *ioq, +int ioprio_class) +{ + ioq->entity.new_ioprio_class = ioprio_class; + ioq->entity.ioprio_changed =1 ; +} + +static inline void elv_ioq_set_ioprio (struct io_queue *ioq,int ioprio) +{ + ioq->entity.new_ioprio = ioprio; + ioq->entity.ioprio_changed =1 ; +} + +static inline void *ioq_sched_queue (struct io_queue *ioq) +{ +if (ioq) +return ioq->sched_queue; +return NULL ; +} + +static inline struct io_group *ioq_to_io_group (struct io_queue *ioq) +{ + return container_of(ioq->entity.sched_data, struct io_group, + sched_data); +} + +extern ssize_t elv_slice_idle_show (struct elevator_queue *q,char *name); +extern ssize_t elv_slice_idle_store (struct elevator_queue *q,const char *name, +size_t count); +extern ssize_t elv_slice_sync_show (struct elevator_queue *q,char *name); +extern ssize_t elv_slice_sync_store (struct elevator_queue *q,const char *name, +size_t count); +extern ssize_t elv_slice_async_show (struct elevator_queue *q,char *name); +extern ssize_t elv_slice_async_store (struct elevator_queue *q,const char *name, +size_t count); + +/* Functions used by elevator.c */ +extern int elv_init_fq_data (struct request_queue *q,struct elevator_queue *e); +extern void elv_exit_fq_data (struct elevator_queue *e); +extern void elv_exit_fq_data_post (struct elevator_queue *e); + +extern void elv_ioq_request_add (struct request_queue *q,struct request *rq); +extern void elv_ioq_request_removed (struct elevator_queue *e, +struct request *rq); +extern void elv_fq_dispatched_request (struct elevator_queue *e, +struct request *rq); + +extern void elv_fq_activate_rq (struct request_queue *q,struct request *rq); +extern void elv_fq_deactivate_rq (struct request_queue *q,struct request *rq); + +extern void elv_ioq_completed_request (struct request_queue *q, +struct request *rq); + +extern void *elv_fq_select_ioq (struct request_queue *q,int force); +extern struct io_queue *rq_ioq (struct request *rq); + +/* Functions used by io schedulers */ +extern void elv_put_ioq (struct io_queue *ioq); +extern void __elv_ioq_slice_expired (struct request_queue *q, +struct io_queue *ioq); +extern int elv_init_ioq (struct elevator_queue *eq,struct io_queue *ioq, +void *sched_queue,int ioprio_class,int ioprio,int is_sync); +extern void elv_schedule_dispatch (struct request_queue *q); +extern int elv_hw_tag (struct elevator_queue *e); +extern void *elv_active_sched_queue (struct elevator_queue *e); +extern int elv_mod_idle_slice_timer (struct elevator_queue *eq, +unsigned long expires); +extern int elv_del_idle_slice_timer (struct elevator_queue *eq); +extern unsigned int elv_get_slice_idle (struct elevator_queue *eq); +extern void *io_group_async_queue_prio (struct io_group *iog,int ioprio_class, +int ioprio); +extern void io_group_set_async_queue (struct io_group *iog,int ioprio_class, +int ioprio,struct io_queue *ioq); +extern struct io_group *io_lookup_io_group_current (struct request_queue *q); +extern int elv_nr_busy_ioq (struct elevator_queue *e); +extern struct io_queue *elv_alloc_ioq (struct request_queue *q,gfp_t gfp_mask); +extern void elv_free_ioq (struct io_queue *ioq); + +#else /* CONFIG_ELV_FAIR_QUEUING */ + +static inline int elv_init_fq_data (struct request_queue *q, +struct elevator_queue *e) +{ +return 0 ; +} + +static inline void elv_exit_fq_data (struct elevator_queue *e) {} +static inline void elv_exit_fq_data_post (struct elevator_queue *e) {} + +static inline void elv_fq_activate_rq (struct request_queue *q, +struct request *rq) +{ +} + +static inline void elv_fq_deactivate_rq (struct request_queue *q, +struct request *rq) +{ +} + +static inline void elv_fq_dispatched_request (struct elevator_queue *e, +struct request *rq) +{ +} + +static inline void elv_ioq_request_removed (struct elevator_queue *e, +struct request *rq) +{ +} + +static inline void elv_ioq_request_add (struct request_queue *q, +struct request *rq) +{ +} + +static inline void elv_ioq_completed_request (struct request_queue *q, +struct request *rq) +{ +} + +static inline void *ioq_sched_queue (struct io_queue *ioq) {return NULL ; } +static inline struct io_queue *rq_ioq (struct request *rq) {return NULL ; } +static inline void *elv_fq_select_ioq (struct request_queue *q,int force) +{ +return NULL ; +} +#endif /* CONFIG_ELV_FAIR_QUEUING */ +#endif /* _BFQ_SCHED_H */ block/elevator.c +35
-11@@ -231,6 +231,9 @@ static struct elevator_queue *elevator_alloc(struct request_queue *q,for (i =0 ; i <ELV_HASH_ENTRIES ; i++)INIT_HLIST_HEAD (&eq->hash[i]); +if (elv_init_fq_data (q, eq)) + goto err; +return eq; err:kfree (eq);@@ -301,9 +304,11 @@ EXPORT_SYMBOL(elevator_init);void elevator_exit (struct elevator_queue *e) {mutex_lock (&e->sysfs_lock); +elv_exit_fq_data (e);if (e->ops->elevator_exit_fn) e->ops->elevator_exit_fn (e); e->ops =NULL ; +elv_exit_fq_data_post (e);mutex_unlock (&e->sysfs_lock);kobject_put (&e->kobj);@@ -314,6 +319,8 @@ static void elv_activate_rq(struct request_queue *q, struct request *rq) {struct elevator_queue *e = q->elevator; +elv_fq_activate_rq (q, rq); +if (e->ops->elevator_activate_req_fn) e->ops->elevator_activate_req_fn (q, rq); }@@ -322,6 +329,8 @@ static void elv_deactivate_rq(struct request_queue *q, struct request *rq) {struct elevator_queue *e = q->elevator; +elv_fq_deactivate_rq (q, rq); +if (e->ops->elevator_deactivate_req_fn) e->ops->elevator_deactivate_req_fn (q, rq); }@@ -446,6 +455,7 @@ void elv_dispatch_sort(struct request_queue *q, struct request *rq)elv_rqhash_del (q, rq); q->nr_sorted--; +elv_fq_dispatched_request (q->elevator, rq); boundary = q->end_sector; stop_flags =REQ_SOFTBARRIER |REQ_HARDBARRIER |REQ_STARTED ;@@ -486,6 +496,7 @@ void elv_dispatch_add_tail(struct request_queue *q, struct request *rq)elv_rqhash_del (q, rq); q->nr_sorted--; +elv_fq_dispatched_request (q->elevator, rq); q->end_sector =rq_end_sector (rq); q->boundary_rq = rq;@@ -553,6 +564,7 @@ void elv_merge_requests(struct request_queue *q, struct request *rq,elv_rqhash_del (q, next); q->nr_sorted--; +elv_ioq_request_removed (e, next); q->last_merge = rq; }@@ -657,12 +669,8 @@ void elv_insert(struct request_queue *q, struct request *rq, int where) q->last_merge = rq; } -/* -* Some ioscheds (cfq) run q->request_fn directly, so -* rq cannot be accessed after calling -* elevator_add_req_fn. -*/ q->elevator->ops->elevator_add_req_fn (q, rq); +elv_ioq_request_add (q, rq);break ;case ELEVATOR_INSERT_REQUEUE :@@ -872,13 +880,12 @@ void elv_dequeue_request(struct request_queue *q, struct request *rq)int elv_queue_empty (struct request_queue *q) { -struct elevator_queue *e = q->elevator; -if (!list_empty (&q->queue_head))return 0 ; -if (e->ops->elevator_queue_empty_fn) -return e->ops->elevator_queue_empty_fn (q); +/* Hopefully nr_sorted works and no need to call queue_empty_fn */ +if (q->nr_sorted) +return 0 ;return 1 ; }@@ -953,8 +960,11 @@ void elv_completed_request(struct request_queue *q, struct request *rq)*/ if (blk_account_rq (rq)) { q->in_flight--; -if (blk_sorted_rq (rq) && e->ops->elevator_completed_req_fn) - e->ops->elevator_completed_req_fn (q, rq); +if (blk_sorted_rq (rq)) { +if (e->ops->elevator_completed_req_fn) + e->ops->elevator_completed_req_fn (q, rq); +elv_ioq_completed_request (q, rq); + } }/* @@ -1242,3 +1252,17 @@ struct request *elv_rb_latter_request(struct request_queue *q,return NULL ; }EXPORT_SYMBOL (elv_rb_latter_request); + +/* Get the io scheduler queue pointer. For cfq, it is stored in rq->ioq*/ +void *elv_get_sched_queue (struct request_queue *q,struct request *rq) +{ +return ioq_sched_queue (rq_ioq (rq)); +} +EXPORT_SYMBOL (elv_get_sched_queue); + +/* Select an ioscheduler queue to dispatch request from. */ +void *elv_select_sched_queue (struct request_queue *q,int force) +{ +return ioq_sched_queue (elv_fq_select_ioq (q, force)); +} +EXPORT_SYMBOL (elv_select_sched_queue);include/linux/blkdev.h +5
-0@@ -245,6 +245,11 @@ struct request {/* for bidi */ struct request *next_rq; + +#ifdef CONFIG_ELV_FAIR_QUEUING +/* io queue request belongs to */ +struct io_queue *ioq; +#endif };static inline unsigned short req_get_ioprio (struct request *req)include/linux/elevator.h +51
-0@@ -2,6 +2,7 @@#define _LINUX_ELEVATOR_H#include <linux/percpu.h> +#include "../../block/elevator-fq.h" #ifdef CONFIG_BLOCK @@ -29,6 +30,18 @@ typedef void (elevator_deactivate_req_fn) (struct request_queue *, struct requestypedef void *(elevator_init_fn ) (struct request_queue *);typedef void (elevator_exit_fn ) (struct elevator_queue *); +#ifdef CONFIG_ELV_FAIR_QUEUING +typedef void (elevator_free_sched_queue_fn ) (struct elevator_queue *,void *); +typedef void (elevator_active_ioq_set_fn ) (struct request_queue *,void *,int ); +typedef void (elevator_active_ioq_reset_fn ) (struct request_queue *,void *); +typedef void (elevator_arm_slice_timer_fn ) (struct request_queue *,void *); +typedef int (elevator_should_preempt_fn ) (struct request_queue *,void *, +struct request *); +typedef int (elevator_update_idle_window_fn ) (struct elevator_queue *,void *, +struct request *); +typedef struct io_queue * (elevator_close_cooperator_fn ) (struct request_queue *, +void *,int probe); +#endif struct elevator_ops {@@ -56,6 +69,17 @@ struct elevator_opselevator_init_fn *elevator_init_fn;elevator_exit_fn *elevator_exit_fn;void (*trim)(struct io_context *); + +#ifdef CONFIG_ELV_FAIR_QUEUING +elevator_free_sched_queue_fn *elevator_free_sched_queue_fn; +elevator_active_ioq_set_fn *elevator_active_ioq_set_fn; +elevator_active_ioq_reset_fn *elevator_active_ioq_reset_fn; + +elevator_arm_slice_timer_fn *elevator_arm_slice_timer_fn; +elevator_should_preempt_fn *elevator_should_preempt_fn; +elevator_update_idle_window_fn *elevator_update_idle_window_fn; +elevator_close_cooperator_fn *elevator_close_cooperator_fn; +#endif };#define ELV_NAME_MAX (16)@@ -76,6 +100,9 @@ struct elevator_typestruct elv_fs_entry *elevator_attrs;char elevator_name[ELV_NAME_MAX ];struct module *elevator_owner; +#ifdef CONFIG_ELV_FAIR_QUEUING +int elevator_features; +#endif };/* @@ -89,6 +116,10 @@ struct elevator_queuestruct elevator_type *elevator_type;struct mutex sysfs_lock;struct hlist_head *hash; +#ifdef CONFIG_ELV_FAIR_QUEUING +/* fair queuing data */ +struct elv_fq_data efqd; +#endif };/* @@ -209,5 +240,25 @@ enum { __val; \ }) +/* iosched can let elevator know their feature set/capability */ +#ifdef CONFIG_ELV_FAIR_QUEUING + +/* iosched wants to use fq logic of elevator layer */ +#define ELV_IOSCHED_NEED_FQ 1 + +static inline int elv_iosched_fair_queuing_enabled (struct elevator_queue *e) +{ +return (e->elevator_type->elevator_features) &ELV_IOSCHED_NEED_FQ ; +} + +#else /* ELV_IOSCHED_FAIR_QUEUING */ + +static inline int elv_iosched_fair_queuing_enabled (struct elevator_queue *e) +{ +return 0 ; +} +#endif /* ELV_IOSCHED_FAIR_QUEUING */ +extern void *elv_get_sched_queue (struct request_queue *q,struct request *rq); +extern void *elv_select_sched_queue (struct request_queue *q,int force);#endif /* CONFIG_BLOCK */ #endif Signature · 1 line
1.6.0.6
Signature · 1 line
Balbir
Re: [PATCH 02/20] io-controller: Common flat fair queuing code in elevaotor layer
Quoted 17 lines
From: Balbir Singh <[email protected]>
Date: Mon, Jun 22, 2009 02:16:12PM +0530Quoted 11 lines · Vivek Goyal
This is common fair queuing code in elevator layer. This is controlled by config option
CONFIG_ELV_FAIR_QUEUING. This patch initially only introduces flat fair queuing support where there is only one group, "root group" and all the tasks belong to root group.This elevator layer changes are backward compatible. That means any ioscheduler using old interfaces will continue to work.
This code is essentially the CFQ code for fair queuing. The primary difference is that flat rounding robin algorithm of CFQ has been replaced with BFQ (WF2Q+).
Quoted 6 lines
The patch is quite long and to be honest requires a long time to review, which I don't mind. I suspect my frequently diverted mind is likely to miss a lot in a big patch like this. Could you consider splitting this further if possible. I think you'll notice the number of reviews will also increase.
This core scheduler part has not changed too much from the bfq patches, so I'll try to answer your questions; Vivek, please correct me where my knowledge is outdated. I preferred to leave out the questions about code that was not in the original patches.
...
Quoted 37 lines
block/elevator-fq.c line 35
Quoted 2 lines
block/elevator-fq.c new file +2
-0+static inline int elv_prio_slice (struct elv_fq_data *efqd,int sync, +unsigned short prio)Why is the return type int and not unsigned int or unsigned long? Can the return value ever be negative?
Quoted 31 lines
block/elevator-fq.c new file +31
-0+{ +const int base_slice = efqd->elv_slice[sync]; + +WARN_ON (prio >=IOPRIO_BE_NR ); + +return base_slice + (base_slice/ELV_SLICE_SCALE * (4 - prio)); +} + +static inline int +elv_prio_to_slice (struct elv_fq_data *efqd,struct io_queue *ioq) +{ +return elv_prio_slice (efqd,elv_ioq_sync (ioq), ioq->entity.ioprio); +} + +/* Mainly the BFQ scheduling code Follows */ + +/* +* Shift for timestamp calculations. This actually limits the maximum +* service allowed in one timestamp delta (small shift values increase it), +* the maximum total weight that can be used for the queues in the system +* (big shift values increase it), and the period of virtual time wraparounds. +*/ +#define WFQ_SERVICE_SHIFT 22 + +/** +* bfq_gt - compare two timestamps. +* @a: first ts. +* @b: second ts. +* +* Return @a > @b, dealing with wrapping correctly. +*/
Quoted 9 lines
block/elevator-fq.c line 71
Quoted 5 lines
block/elevator-fq.c new file +5
-0+static inline int bfq_gt (bfq_timestamp_t a,bfq_timestamp_t b) +{ +return (s64)(a - b) >0 ; +} +a and b are of type u64, but cast to s64 to deal with wrapping? Correct?
yes
Quoted 5 lines
Quoted 5 lines
+/**
+ *bfq_delta- map service into the virtual time domain.
+ * @service: amount of service.
+ * @weight: scale factor.
+ */
Quoted 9 lines
block/elevator-fq.c line 81
Quoted 5 lines
block/elevator-fq.c new file +5
-0+static inline bfq_timestamp_t bfq_delta (bfq_service_t service, +bfq_weight_t weight) +{ +bfq_timestamp_t d = (bfq_timestamp_t )service <<WFQ_SERVICE_SHIFT ; +Why is the case required? Does the compiler complain? service is already of the correct type.
service is unsigned long, so it can be 32 bits in 32 bit machines, while timestamps are always u64, so I think we need the cast.
Quoted 6 lines
block/elevator-fq.c line 82
Quoted 1 line
block/elevator-fq.c new file +1
-0+do_div (d, weight);On a 64 system both d and weight are 64 bit, but on a 32 bit system weight is 32 bits.
do_divexpects a 64 bit dividend and 32 bit divisor
- no?
yes. here the situation is that we actually don't care about the type of weight, as long as it can contain a 32 bit value, and weights should never reach near the 2^32 boundary, otherwise we're prone to any kind of numerical error. there are no problems with weight being u32.
Quoted 10 lines
Quoted 10 lines
block/elevator-fq.c new file +10
-0+return d; +} + +/** +* bfq_calc_finish - assign the finish time to an entity. +* @entity: the entity to act upon. +* @service: the service to be charged to the entity. +*/ +static inline void bfq_calc_finish (struct io_entity *entity, +bfq_service_t service)
Quoted 9 lines
block/elevator-fq.c line 97
Quoted 5 lines
block/elevator-fq.c new file +5
-0+{ +BUG_ON (entity->weight ==0 ); + + entity->finish = entity->start +bfq_delta (service, entity->weight); +}Should we
BUG_ON(entity->finish==entity->start)? Or is that expected when the entity has no service time left.
bfq_calc_finish() is used in two cases:
1) we need to resync the finish time with the service received by an entity
2) we need to assign a new finish time to an entity when it's enqueued
with preemptions 1) can happen with service = 0, and we need to reset the finish time to the start time (depending on how preemptions are implemented), so in this case we'd have a false positive (leading to a crashed system :) ).
Quoted 40 lines
Quoted 27 lines
block/elevator-fq.c new file +27
-0+ +static inline struct io_queue *io_entity_to_ioq (struct io_entity *entity) +{ +struct io_queue *ioq =NULL ; + +BUG_ON (entity ==NULL ); +if (entity->my_sched_data ==NULL ) + ioq = container_of(entity, struct io_queue, entity); +return ioq; +} + +/** +* bfq_entity_of - get an entity from a node. +* @node: the node field of the entity. +* +* Convert a node pointer to the relative entity. This is used only +* to simplify the logic of some functions and not as the generic +* conversion mechanism because, e.g., in the tree walking functions, +* the check for a %NULL value would be redundant. +*/ +static inline struct io_entity *bfq_entity_of (struct rb_node *node) +{ +struct io_entity *entity =NULL ; + +if (node !=NULL ) + entity = rb_entry(node, struct io_entity, rb_node); +block/elevator-fq.c line 134
Quoted 10 lines
block/elevator-fq.c new file +10
-0+return entity; +} + +/** +* bfq_extract - remove an entity from a tree. +* @root: the tree root. +* @entity: the entity to remove. +*/ +static inline void bfq_extract (struct rb_root *root,struct io_entity *entity) +{Extract is not common terminology, why not use
bfq_remove()?
Quoted 7 lines
block/elevator-fq.c line 138
Quoted 4 lines
block/elevator-fq.c new file +4
-0+BUG_ON (entity->tree != root); + + entity->tree =NULL ; +rb_erase (&entity->rb_node, root);Don't you want to make
entity->tree= NULL afterrb_erase?
this code assumes to be executed under spinlock, so order doesn't really matter (tree is not affected by rb_erase(), it is a bfq private field).
Quoted 10 lines
Quoted 10 lines
block/elevator-fq.c new file +10
-0+} + +/** +* bfq_idle_extract - extract an entity from the idle tree. +* @st: the service tree of the owning @entity. +* @entity: the entity being removed. +*/ +static void bfq_idle_extract (struct io_service_tree *st, +struct io_entity *entity) +{
Quoted 9 lines
block/elevator-fq.c line 154
Quoted 6 lines
block/elevator-fq.c new file +6
-0+struct rb_node *next; + +BUG_ON (entity->tree != &st->idle); + +if (entity == st->first_idle) { + next =rb_next (&entity->rb_node);What happens if next is NULL?
the bfq_entity_of() call below returns NULL
Quoted 8 lines
block/elevator-fq.c line 159
Quoted 5 lines
block/elevator-fq.c new file +5
-0+ st->first_idle =bfq_entity_of (next); + } + +if (entity == st->last_idle) { + next =rb_prev (&entity->rb_node);What happens if next is NULL?
same as above
Quoted 55 lines
Quoted 55 lines
block/elevator-fq.c new file +55
-0+ st->last_idle =bfq_entity_of (next); + } + +bfq_extract (&st->idle, entity); +} + +/** +* bfq_insert - generic tree insertion. +* @root: tree root. +* @entity: entity to insert. +* +* This is used for the idle and the active tree, since they are both +* ordered by finish time. +*/ +static void bfq_insert (struct rb_root *root,struct io_entity *entity) +{ +struct io_entity *entry; +struct rb_node **node = &root->rb_node; +struct rb_node *parent =NULL ; + +BUG_ON (entity->tree !=NULL ); + +while (*node !=NULL ) { + parent = *node; + entry = rb_entry(parent, struct io_entity, rb_node); + +if (bfq_gt (entry->finish, entity->finish)) + node = &parent->rb_left; +else + node = &parent->rb_right; + } + +rb_link_node (&entity->rb_node, parent, node); +rb_insert_color (&entity->rb_node, root); + + entity->tree = root; +} + +/** +* bfq_update_min - update the min_start field of a entity. +* @entity: the entity to update. +* @node: one of its children. +* +* This function is called when @entity may store an invalid value for +* min_start due to updates to the active tree. The function assumes +* that the subtree rooted at @node (which may be its left or its right +* child) has a valid min_start value. +*/ +static inline void bfq_update_min (struct io_entity *entity, +struct rb_node *node) +{ +struct io_entity *child; + +if (node !=NULL ) { + child = rb_entry(node, struct io_entity, rb_node);
Quoted 9 lines
block/elevator-fq.c line 218
Quoted 4 lines
block/elevator-fq.c new file +4
-0+if (bfq_gt (entity->min_start, child->min_start)) + entity->min_start = child->min_start; + } +}So.. we check to see if child's
min_timeis lesser than the root entities or node entities and set it to the minimum of the two? Can you usemin_there?
no, it would not deal with wraparound correctly
Quoted 13 lines
Quoted 13 lines
block/elevator-fq.c new file +13
-0+ +/** +* bfq_update_active_node - recalculate min_start. +* @node: the node to update. +* +* @node may have changed position or one of its children may have moved, +* this function updates its min_start value. The left and right subtrees +* are assumed to hold a correct min_start value. +*/ +static inline void bfq_update_active_node (struct rb_node *node) +{ + struct io_entity *entity = rb_entry(node, struct io_entity, rb_node); +
Quoted 9 lines
block/elevator-fq.c line 235
Quoted 4 lines
block/elevator-fq.c new file +4
-0+ entity->min_start = entity->start; +bfq_update_min (entity, node->rb_right); +bfq_update_min (entity, node->rb_left); +}I don't like this every much, we set the
min_timetwice, this can be easily optimized to look at both left and right child and pick the minimum.
it's a minimum between three values (the ->start fields of the two children and of the node itself), you cannot be sure it will be set twice
Quoted 26 lines
Quoted 26 lines
block/elevator-fq.c new file +26
-0+ +/** +* bfq_update_active_tree - update min_start for the whole active tree. +* @node: the starting node. +* +* @node must be the deepest modified node after an update. This function +* updates its min_start using the values held by its children, assuming +* that they did not change, and then updates all the nodes that may have +* changed in the path to the root. The only nodes that may have changed +* are the ones in the path or their siblings. +*/ +static void bfq_update_active_tree (struct rb_node *node) +{ +struct rb_node *parent; + +up: +bfq_update_active_node (node); + + parent =rb_parent (node); +if (parent ==NULL ) +return ; + +if (node == parent->rb_left && parent->rb_right !=NULL ) +bfq_update_active_node (parent->rb_right); +else if (parent->rb_left !=NULL ) +bfq_update_active_node (parent->rb_left);
Quoted 9 lines
block/elevator-fq.c line 266
Quoted 5 lines
block/elevator-fq.c new file +5
-0+ + node = parent; + goto up; +} +For these functions, take a look at the walk function in the group scheduler that does
update_shares
are you sure? AFAICT walk_tg_tree() walks all over the tree, this just walks a single path to a node up to the root, I don't see what the two have in common.
in the original patches we cited (among the others):
http://www.cs.berkeley.edu/~istoica/papers/eevdf-tr-95.pdf
which contains a description of the algorithm.
Quoted 59 lines
Quoted 59 lines
block/elevator-fq.c new file +59
-0+/** +* bfq_active_insert - insert an entity in the active tree of its group/device. +* @st: the service tree of the entity. +* @entity: the entity being inserted. +* +* The active tree is ordered by finish time, but an extra key is kept +* per each node, containing the minimum value for the start times of +* its children (and the node itself), so it's possible to search for +* the eligible node with the lowest finish time in logarithmic time. +*/ +static void bfq_active_insert (struct io_service_tree *st, +struct io_entity *entity) +{ +struct rb_node *node = &entity->rb_node; + +bfq_insert (&st->active, entity); + +if (node->rb_left !=NULL ) + node = node->rb_left; +else if (node->rb_right !=NULL ) + node = node->rb_right; + +bfq_update_active_tree (node); +} + +/** +* bfq_ioprio_to_weight - calc a weight from an ioprio. +* @ioprio: the ioprio value to convert. +*/ +static bfq_weight_t bfq_ioprio_to_weight (int ioprio) +{ +WARN_ON (ioprio <0 || ioprio >=IOPRIO_BE_NR ); +return IOPRIO_BE_NR - ioprio; +} + +void bfq_get_entity (struct io_entity *entity) +{ +struct io_queue *ioq =io_entity_to_ioq (entity); + +if (ioq) +elv_get_ioq (ioq); +} + +void bfq_init_entity (struct io_entity *entity,struct io_group *iog) +{ + entity->ioprio = entity->new_ioprio; + entity->ioprio_class = entity->new_ioprio_class; + entity->sched_data = &iog->sched_data; +} + +/** +* bfq_find_deepest - find the deepest node that an extraction can modify. +* @node: the node being removed. +* +* Do the first step of an extraction in an rb tree, looking for the +* node that will replace @node, and returning the deepest node that +* the following modifications to the tree can touch. If @node is the +* last node in the tree return %NULL. +*/
Quoted 9 lines
block/elevator-fq.c line 331
Quoted 6 lines
block/elevator-fq.c new file +6
-0+static struct rb_node *bfq_find_deepest (struct rb_node *node) +{ +struct rb_node *deepest; + +if (node->rb_right ==NULL && node->rb_left ==NULL ) + deepest =rb_parent (node);Why is the parent the deepest? Shouldn't node be the deepest?
this is related to how the RB tree is updated (see below)
Quoted 9 lines
Quoted 9 lines
block/elevator-fq.c new file +9
-0+else if (node->rb_right ==NULL ) + deepest = node->rb_left; +else if (node->rb_left ==NULL ) + deepest = node->rb_right; +else { + deepest =rb_next (node); +if (deepest->rb_right !=NULL ) + deepest = deepest->rb_right; +else if (rb_parent (deepest) != node)
Quoted 9 lines
block/elevator-fq.c line 345
Quoted 5 lines
block/elevator-fq.c new file +5
-0+ deepest =rb_parent (deepest); + } + +return deepest; +}The function is not clear, could you please define deepest node better?
according to the paper cited above, we need to update the min_start value on the path from the deepest node modified by the extraction up to the root. this function tries to consider all the cases of RB extraction, looking for the deepest node that (after all the rotations etc.) will need an update to min_start. one interesting property of RB trees is that this can be done in O(log N) because there is a single path that needs to be updated.
Quoted 13 lines
Quoted 13 lines
block/elevator-fq.c new file +13
-0+ +/** +* bfq_active_extract - remove an entity from the active tree. +* @st: the service_tree containing the tree. +* @entity: the entity being removed. +*/ +static void bfq_active_extract (struct io_service_tree *st, +struct io_entity *entity) +{ +struct rb_node *node; + + node =bfq_find_deepest (&entity->rb_node); +bfq_extract (&st->active, entity);
Quoted 9 lines
block/elevator-fq.c line 363
Quoted 5 lines
block/elevator-fq.c new file +5
-0+ +if (node !=NULL ) +bfq_update_active_tree (node); +} +Just to check my understanding, every time an active node is added/removed, we update the
min_timeof the entire tree.
yes, but only O(log N) nodes need to be updated
Quoted 334 lines
Quoted 334 lines
block/elevator-fq.c new file +334
-0+/** +* bfq_idle_insert - insert an entity into the idle tree. +* @st: the service tree containing the tree. +* @entity: the entity to insert. +*/ +static void bfq_idle_insert (struct io_service_tree *st, +struct io_entity *entity) +{ +struct io_entity *first_idle = st->first_idle; +struct io_entity *last_idle = st->last_idle; + +if (first_idle ==NULL ||bfq_gt (first_idle->finish, entity->finish)) + st->first_idle = entity; +if (last_idle ==NULL ||bfq_gt (entity->finish, last_idle->finish)) + st->last_idle = entity; + +bfq_insert (&st->idle, entity); +} + +/** +* bfq_forget_entity - remove an entity from the wfq trees. +* @st: the service tree. +* @entity: the entity being removed. +* +* Update the device status and forget everything about @entity, putting +* the device reference to it, if it is a queue. Entities belonging to +* groups are not refcounted. +*/ +static void bfq_forget_entity (struct io_service_tree *st, +struct io_entity *entity) +{ +struct io_queue *ioq =NULL ; + +BUG_ON (!entity->on_st); + entity->on_st =0 ; + st->wsum -= entity->weight; + ioq =io_entity_to_ioq (entity); +if (!ioq) +return ; +elv_put_ioq (ioq); +} + +/** +* bfq_put_idle_entity - release the idle tree ref of an entity. +* @st: service tree for the entity. +* @entity: the entity being released. +*/ +void bfq_put_idle_entity (struct io_service_tree *st, +struct io_entity *entity) +{ +bfq_idle_extract (st, entity); +bfq_forget_entity (st, entity); +} + +/** +* bfq_forget_idle - update the idle tree if necessary. +* @st: the service tree to act upon. +* +* To preserve the global O(log N) complexity we only remove one entry here; +* as the idle tree will not grow indefinitely this can be done safely. +*/ +void bfq_forget_idle (struct io_service_tree *st) +{ +struct io_entity *first_idle = st->first_idle; +struct io_entity *last_idle = st->last_idle; + +if (RB_EMPTY_ROOT (&st->active) && last_idle !=NULL && + !bfq_gt (last_idle->finish, st->vtime)) { +/* +* Active tree is empty. Pull back vtime to finish time of +* last idle entity on idle tree. +* Rational seems to be that it reduces the possibility of +* vtime wraparound (bfq_gt(V-F) < 0). +*/ + st->vtime = last_idle->finish; + } + +if (first_idle !=NULL && !bfq_gt (first_idle->finish, st->vtime)) +bfq_put_idle_entity (st, first_idle); +} + + +static struct io_service_tree * +__bfq_entity_update_prio (struct io_service_tree *old_st, +struct io_entity *entity) +{ +struct io_service_tree *new_st = old_st; +struct io_queue *ioq =io_entity_to_ioq (entity); + +if (entity->ioprio_changed) { + entity->ioprio = entity->new_ioprio; + entity->ioprio_class = entity->new_ioprio_class; + entity->ioprio_changed =0 ; + +/* +* Also update the scaled budget for ioq. Group will get the +* updated budget once ioq is selected to run next. +*/ +if (ioq) { +struct elv_fq_data *efqd = ioq->efqd; + entity->budget =elv_prio_to_slice (efqd, ioq); + } + + old_st->wsum -= entity->weight; + entity->weight =bfq_ioprio_to_weight (entity->ioprio); + +/* +* NOTE: here we may be changing the weight too early, +* this will cause unfairness. The correct approach +* would have required additional complexity to defer +* weight changes to the proper time instants (i.e., +* when entity->finish <= old_st->vtime). +*/ + new_st =io_entity_service_tree (entity); + new_st->wsum += entity->weight; + +if (new_st != old_st) + entity->start = new_st->vtime; + } + +return new_st; +} + +/** +* __bfq_activate_entity - activate an entity. +* @entity: the entity being activated. +* +* Called whenever an entity is activated, i.e., it is not active and one +* of its children receives a new request, or has to be reactivated due to +* budget exhaustion. It uses the current budget of the entity (and the +* service received if @entity is active) of the queue to calculate its +* timestamps. +*/ +static void __bfq_activate_entity (struct io_entity *entity,int add_front) +{ +struct io_sched_data *sd = entity->sched_data; +struct io_service_tree *st =io_entity_service_tree (entity); + +if (entity == sd->active_entity) { +BUG_ON (entity->tree !=NULL ); +/* +* If we are requeueing the current entity we have +* to take care of not charging to it service it has +* not received. +*/ +bfq_calc_finish (entity, entity->service); + entity->start = entity->finish; + sd->active_entity =NULL ; + }else if (entity->tree == &st->active) { +/* +* Requeueing an entity due to a change of some +* next_active entity below it. We reuse the old +* start time. +*/ +bfq_active_extract (st, entity); + }else if (entity->tree == &st->idle) { +/* +* Must be on the idle tree, bfq_idle_extract() will +* check for that. +*/ +bfq_idle_extract (st, entity); + entity->start =bfq_gt (st->vtime, entity->finish) ? + st->vtime : entity->finish; + }else { +/* +* The finish time of the entity may be invalid, and +* it is in the past for sure, otherwise the queue +* would have been on the idle tree. +*/ + entity->start = st->vtime; + st->wsum += entity->weight; +bfq_get_entity (entity); + +BUG_ON (entity->on_st); + entity->on_st =1 ; + } + + st =__bfq_entity_update_prio (st, entity); +/* +* This is to emulate cfq like functionality where preemption can +* happen with-in same class, like sync queue preempting async queue +* May be this is not a very good idea from fairness point of view +* as preempting queue gains share. Keeping it for now. +*/ +if (add_front) { +struct io_entity *next_entity; + +/* +* Determine the entity which will be dispatched next +* Use sd->next_active once hierarchical patch is applied +*/ + next_entity =bfq_lookup_next_entity (sd,0 ); + +if (next_entity && next_entity != entity) { +struct io_service_tree *new_st; +bfq_timestamp_t delta; + + new_st =io_entity_service_tree (next_entity); + +/* +* At this point, both entities should belong to +* same service tree as cross service tree preemption +* is automatically taken care by algorithm +*/ +BUG_ON (new_st != st); + entity->finish = next_entity->finish -1 ; + delta =bfq_delta (entity->budget, entity->weight); + entity->start = entity->finish - delta; +if (bfq_gt (entity->start, st->vtime)) + entity->start = st->vtime; + } + }else { +bfq_calc_finish (entity, entity->budget); + } +bfq_active_insert (st, entity); +} + +/** +* bfq_activate_entity - activate an entity. +* @entity: the entity to activate. +*/ +void bfq_activate_entity (struct io_entity *entity,int add_front) +{ +__bfq_activate_entity (entity, add_front); +} + +/** +* __bfq_deactivate_entity - deactivate an entity from its service tree. +* @entity: the entity to deactivate. +* @requeue: if false, the entity will not be put into the idle tree. +* +* Deactivate an entity, independently from its previous state. If the +* entity was not on a service tree just return, otherwise if it is on +* any scheduler tree, extract it from that tree, and if necessary +* and if the caller did not specify @requeue, put it on the idle tree. +* +*/ +int __bfq_deactivate_entity (struct io_entity *entity,int requeue) +{ +struct io_sched_data *sd = entity->sched_data; +struct io_service_tree *st =io_entity_service_tree (entity); +int was_active = entity == sd->active_entity; +int ret =0 ; + +if (!entity->on_st) +return 0 ; + +BUG_ON (was_active && entity->tree !=NULL ); + +if (was_active) { +bfq_calc_finish (entity, entity->service); + sd->active_entity =NULL ; + }else if (entity->tree == &st->active) +bfq_active_extract (st, entity); +else if (entity->tree == &st->idle) +bfq_idle_extract (st, entity); +else if (entity->tree !=NULL ) +BUG (); + +if (!requeue || !bfq_gt (entity->finish, st->vtime)) +bfq_forget_entity (st, entity); +else +bfq_idle_insert (st, entity); + +BUG_ON (sd->active_entity == entity); + +return ret; +} + +/** +* bfq_deactivate_entity - deactivate an entity. +* @entity: the entity to deactivate. +* @requeue: true if the entity can be put on the idle tree +*/ +void bfq_deactivate_entity (struct io_entity *entity,int requeue) +{ +__bfq_deactivate_entity (entity, requeue); +} + +/** +* bfq_update_vtime - update vtime if necessary. +* @st: the service tree to act upon. +* +* If necessary update the service tree vtime to have at least one +* eligible entity, skipping to its start time. Assumes that the +* active tree of the device is not empty. +* +* NOTE: this hierarchical implementation updates vtimes quite often, +* we may end up with reactivated tasks getting timestamps after a +* vtime skip done because we needed a ->first_active entity on some +* intermediate node. +*/ +static void bfq_update_vtime (struct io_service_tree *st) +{ +struct io_entity *entry; +struct rb_node *node = st->active.rb_node; + + entry = rb_entry(node, struct io_entity, rb_node); +if (bfq_gt (entry->min_start, st->vtime)) { + st->vtime = entry->min_start; +bfq_forget_idle (st); + } +} + +/** +* bfq_first_active - find the eligible entity with the smallest finish time +* @st: the service tree to select from. +* +* This function searches the first schedulable entity, starting from the +* root of the tree and going on the left every time on this side there is +* a subtree with at least one eligible (start <= vtime) entity. The path +* on the right is followed only if a) the left subtree contains no eligible +* entities and b) no eligible entity has been found yet. +*/ +static struct io_entity *bfq_first_active_entity (struct io_service_tree *st) +{ +struct io_entity *entry, *first =NULL ; +struct rb_node *node = st->active.rb_node; + +while (node !=NULL ) { + entry = rb_entry(node, struct io_entity, rb_node); +left: +if (!bfq_gt (entry->start, st->vtime)) + first = entry; + +BUG_ON (bfq_gt (entry->min_start, st->vtime)); + +if (node->rb_left !=NULL ) { + entry =rb_entry (node->rb_left, + struct io_entity, rb_node); +if (!bfq_gt (entry->min_start, st->vtime)) { + node = node->rb_left; + goto left; + }
Quoted 9 lines
block/elevator-fq.c line 701
Quoted 4 lines
block/elevator-fq.c new file +4
-0+ } +if (first !=NULL ) +break ; + node = node->rb_right;Please help me understand this, we sort the tree by finish time, but search by vtime,
start_time. The worst case could easily be O(N), right?
no, (again, the full answer is in the paper); the nice property of min_start is that it partitions the tree in two regions, one with eligible entities and one without any of them. once we know that there is one eligible entity (checking the min_start at the root) we can find the node i with min(F_i) subject to S_i < V walking down a single path from the root to the leftmost eligible entity. (we need to go to the right only if the subtree on the left contains no eligible entities at all.) since the RB tree is balanced this can be done in O(log N).
Quoted 69 lines
Quoted 69 lines
block/elevator-fq.c new file +69
-0+ } + +BUG_ON (first ==NULL && !RB_EMPTY_ROOT (&st->active)); +return first; +} + +/** +* __bfq_lookup_next_entity - return the first eligible entity in @st. +* @st: the service tree. +* +* Update the virtual time in @st and return the first eligible entity +* it contains. +*/ +static struct io_entity *__bfq_lookup_next_entity (struct io_service_tree *st) +{ +struct io_entity *entity; + +if (RB_EMPTY_ROOT (&st->active)) +return NULL ; + +bfq_update_vtime (st); + entity =bfq_first_active_entity (st); +BUG_ON (bfq_gt (entity->start, st->vtime)); + +return entity; +} + +/** +* bfq_lookup_next_entity - return the first eligible entity in @sd. +* @sd: the sched_data. +* @extract: if true the returned entity will be also extracted from @sd. +* +* NOTE: since we cache the next_active entity at each level of the +* hierarchy, the complexity of the lookup can be decreased with +* absolutely no effort just returning the cached next_active value; +* we prefer to do full lookups to test the consistency of * the data +* structures. +*/ +struct io_entity *bfq_lookup_next_entity (struct io_sched_data *sd, +int extract) +{ +struct io_service_tree *st = sd->service_tree; +struct io_entity *entity; +int i; + +/* +* We should not call lookup when an entity is active, as doing lookup +* can result in an erroneous vtime jump. +*/ +BUG_ON (sd->active_entity !=NULL ); + +for (i =0 ; i <IO_IOPRIO_CLASSES ; i++, st++) { + entity =__bfq_lookup_next_entity (st); +if (entity !=NULL ) { +if (extract) { +bfq_active_extract (st, entity); + sd->active_entity = entity; + } +break ; + } + } + +return entity; +} + +void entity_served (struct io_entity *entity,bfq_service_t served) +{ +struct io_service_tree *st; +
Quoted 9 lines
block/elevator-fq.c line 775
Quoted 5 lines
block/elevator-fq.c new file +5
-0+ st =io_entity_service_tree (entity); + entity->service += served; +BUG_ON (st->wsum ==0 ); + st->vtime +=bfq_delta (served, st->wsum); +bfq_forget_idle (st);Forget idle checks to see if the
st->vtime>first_idle->finish, if so it pushes thefirst_idleto later, right?
yes, updating the weight sum accordingly
Quoted 182 lines
Quoted 182 lines
block/elevator-fq.c new file +182
-0+} + +/** +* bfq_flush_idle_tree - deactivate any entity on the idle tree of @st. +* @st: the service tree being flushed. +*/ +void io_flush_idle_tree (struct io_service_tree *st) +{ +struct io_entity *entity = st->first_idle; + +for (; entity !=NULL ; entity = st->first_idle) +__bfq_deactivate_entity (entity,0 ); +} + +/* Elevator fair queuing function */ +struct io_queue *rq_ioq (struct request *rq) +{ +return rq->ioq; +} + +static inline struct io_queue *elv_active_ioq (struct elevator_queue *e) +{ +return e->efqd.active_queue; +} + +void *elv_active_sched_queue (struct elevator_queue *e) +{ +return ioq_sched_queue (elv_active_ioq (e)); +} +EXPORT_SYMBOL (elv_active_sched_queue); + +int elv_nr_busy_ioq (struct elevator_queue *e) +{ +return e->efqd.busy_queues; +} +EXPORT_SYMBOL (elv_nr_busy_ioq); + +int elv_hw_tag (struct elevator_queue *e) +{ +return e->efqd.hw_tag; +} +EXPORT_SYMBOL (elv_hw_tag); + +/* Helper functions for operating on elevator idle slice timer */ +int elv_mod_idle_slice_timer (struct elevator_queue *eq,unsigned long expires) +{ +struct elv_fq_data *efqd = &eq->efqd; + +return mod_timer (&efqd->idle_slice_timer, expires); +} +EXPORT_SYMBOL (elv_mod_idle_slice_timer); + +int elv_del_idle_slice_timer (struct elevator_queue *eq) +{ +struct elv_fq_data *efqd = &eq->efqd; + +return del_timer (&efqd->idle_slice_timer); +} +EXPORT_SYMBOL (elv_del_idle_slice_timer); + +unsigned int elv_get_slice_idle (struct elevator_queue *eq) +{ +return eq->efqd.elv_slice_idle; +} +EXPORT_SYMBOL (elv_get_slice_idle); + +void elv_ioq_served (struct io_queue *ioq,bfq_service_t served) +{ +entity_served (&ioq->entity, served); +} + +/* Tells whether ioq is queued in root group or not */ +static inline int is_root_group_ioq (struct request_queue *q, +struct io_queue *ioq) +{ +struct elv_fq_data *efqd = &q->elevator->efqd; + +return (ioq->entity.sched_data == &efqd->root_group->sched_data); +} + +/* +* sysfs parts below --> +*/ +static ssize_t +elv_var_show (unsigned int var,char *page) +{ +return sprintf (page,"%d\n" , var); +} + +static ssize_t +elv_var_store (unsigned int *var,const char *page,size_t count) +{ +char *p = (char *) page; + + *var =simple_strtoul (p, &p,10 ); +return count; +} + +#define SHOW_FUNCTION (__FUNC, __VAR, __CONV) \ +ssize_t __FUNC(struct elevator_queue *e, char *page) \ +{ \ + struct elv_fq_data *efqd = &e->efqd; \ + unsigned int __data = __VAR; \ + if (__CONV) \ + __data = jiffies_to_msecs(__data); \ + return elv_var_show(__data, (page)); \ +} +SHOW_FUNCTION (elv_slice_idle_show, efqd->elv_slice_idle,1 ); +EXPORT_SYMBOL (elv_slice_idle_show); +SHOW_FUNCTION (elv_slice_sync_show, efqd->elv_slice[1 ],1 ); +EXPORT_SYMBOL (elv_slice_sync_show); +SHOW_FUNCTION (elv_slice_async_show, efqd->elv_slice[0 ],1 ); +EXPORT_SYMBOL (elv_slice_async_show); +#undef SHOW_FUNCTION + +#define STORE_FUNCTION (__FUNC, __PTR,MIN ,MAX , __CONV) \ +ssize_t __FUNC(struct elevator_queue *e, const char *page, size_t count)\ +{ \ + struct elv_fq_data *efqd = &e->efqd; \ + unsigned int __data; \ + int ret = elv_var_store(&__data, (page), count); \ + if (__data < (MIN)) \ + __data = (MIN); \ + else if (__data > (MAX)) \ + __data = (MAX); \ + if (__CONV) \ + *(__PTR) = msecs_to_jiffies(__data); \ + else \ + *(__PTR) = __data; \ + return ret; \ +} +STORE_FUNCTION (elv_slice_idle_store, &efqd->elv_slice_idle,0 ,UINT_MAX ,1 ); +EXPORT_SYMBOL (elv_slice_idle_store); +STORE_FUNCTION (elv_slice_sync_store, &efqd->elv_slice[1 ],1 ,UINT_MAX ,1 ); +EXPORT_SYMBOL (elv_slice_sync_store); +STORE_FUNCTION (elv_slice_async_store, &efqd->elv_slice[0 ],1 ,UINT_MAX ,1 ); +EXPORT_SYMBOL (elv_slice_async_store); +#undef STORE_FUNCTION + +void elv_schedule_dispatch (struct request_queue *q) +{ +struct elv_fq_data *efqd = &q->elevator->efqd; + +if (elv_nr_busy_ioq (q->elevator)) { +elv_log (efqd,"schedule dispatch" ); +kblockd_schedule_work (efqd->queue, &efqd->unplug_work); + } +} +EXPORT_SYMBOL (elv_schedule_dispatch); + +void elv_kick_queue (struct work_struct *work) +{ +struct elv_fq_data *efqd = + container_of(work, struct elv_fq_data, unplug_work); +struct request_queue *q = efqd->queue; +unsigned long flags; + +spin_lock_irqsave (q->queue_lock, flags); +blk_start_queueing (q); +spin_unlock_irqrestore (q->queue_lock, flags); +} + +void elv_shutdown_timer_wq (struct elevator_queue *e) +{ +del_timer_sync (&e->efqd.idle_slice_timer); +cancel_work_sync (&e->efqd.unplug_work); +} +EXPORT_SYMBOL (elv_shutdown_timer_wq); + +void elv_ioq_set_prio_slice (struct request_queue *q,struct io_queue *ioq) +{ +struct elv_fq_data *efqd = &q->elevator->efqd; + + ioq->slice_end = jiffies + ioq->entity.budget; +elv_log_ioq (efqd, ioq,"set_slice=%lu" , ioq->entity.budget); +} + +static void elv_ioq_update_io_thinktime (struct io_queue *ioq) +{ +struct elv_fq_data *efqd = ioq->efqd; +unsigned long elapsed = jiffies - ioq->last_end_request; +unsigned long ttime =min (elapsed,2UL * efqd->elv_slice_idle);
Quoted 9 lines
block/elevator-fq.c line 962
Quoted 5 lines
block/elevator-fq.c new file +5
-0+ + ioq->ttime_samples = (7 *ioq->ttime_samples +256 ) /8 ; + ioq->ttime_total = (7 *ioq->ttime_total +256 *ttime) /8 ; + ioq->ttime_mean = (ioq->ttime_total +128 ) / ioq->ttime_samples; +}Not sure I understand the magical 7, 8, 2, 128 and 256. Please help me understand the algorithm.
this came from cfq, it's a variation of an exponential moving average, with ttime_samples used to scale the average value.
Quoted 1,083 lines
Quoted 68 lines
block/elevator-fq.c new file +68
-0+ +/* +* Disable idle window if the process thinks too long. +* This idle flag can also be updated by io scheduler. +*/ +static void elv_ioq_update_idle_window (struct elevator_queue *eq, +struct io_queue *ioq,struct request *rq) +{ +int old_idle, enable_idle; +struct elv_fq_data *efqd = ioq->efqd; + +/* +* Don't idle for async or idle io prio class +*/ +if (!elv_ioq_sync (ioq) ||elv_ioq_class_idle (ioq)) +return ; + + enable_idle = old_idle =elv_ioq_idle_window (ioq); + +if (!efqd->elv_slice_idle) + enable_idle =0 ; +else if (ioq_sample_valid (ioq->ttime_samples)) { +if (ioq->ttime_mean > efqd->elv_slice_idle) + enable_idle =0 ; +else + enable_idle =1 ; + } + +/* +* From think time perspective idle should be enabled. Check with +* io scheduler if it wants to disable idling based on additional +* considrations like seek pattern. +*/ +if (enable_idle) { +if (eq->ops->elevator_update_idle_window_fn) + enable_idle = eq->ops->elevator_update_idle_window_fn ( + eq, ioq->sched_queue, rq); +if (!enable_idle) +elv_log_ioq (efqd, ioq,"iosched disabled idle" ); + } + +if (old_idle != enable_idle) { +elv_log_ioq (efqd, ioq,"idle=%d" , enable_idle); +if (enable_idle) +elv_mark_ioq_idle_window (ioq); +else +elv_clear_ioq_idle_window (ioq); + } +} + +struct io_queue *elv_alloc_ioq (struct request_queue *q,gfp_t gfp_mask) +{ +struct io_queue *ioq =NULL ; + + ioq =kmem_cache_alloc_node (elv_ioq_pool, gfp_mask, q->node); +return ioq; +} +EXPORT_SYMBOL (elv_alloc_ioq); + +void elv_free_ioq (struct io_queue *ioq) +{ +kmem_cache_free (elv_ioq_pool, ioq); +} +EXPORT_SYMBOL (elv_free_ioq); + +int elv_init_ioq (struct elevator_queue *eq,struct io_queue *ioq, +void *sched_queue,int ioprio_class,int ioprio, +int is_sync)block/elevator-fq.c line 1040
Quoted 10 lines
block/elevator-fq.c new file +10
-0+{ +struct elv_fq_data *efqd = &eq->efqd; +struct io_group *iog =io_lookup_io_group_current (efqd->queue); + +RB_CLEAR_NODE (&ioq->entity.rb_node); +atomic_set (&ioq->ref,0 ); + ioq->efqd = efqd; +elv_ioq_set_ioprio_class (ioq, ioprio_class); +elv_ioq_set_ioprio (ioq, ioprio); + ioq->pid = current->pid;Is pid used for cgroup association later? I don't see why we save the pid otherwise? If yes, why not store the cgroup of the
current->pid?Quoted 95 lines
block/elevator-fq.c new file +95
-0+ ioq->sched_queue = sched_queue; +if (is_sync && !elv_ioq_class_idle (ioq)) +elv_mark_ioq_idle_window (ioq); +bfq_init_entity (&ioq->entity, iog); + ioq->entity.budget =elv_prio_to_slice (efqd, ioq); +if (is_sync) + ioq->last_end_request = jiffies; + +return 0 ; +} +EXPORT_SYMBOL (elv_init_ioq); + +void elv_put_ioq (struct io_queue *ioq) +{ +struct elv_fq_data *efqd = ioq->efqd; + struct elevator_queue *e = container_of(efqd, struct elevator_queue, + efqd); + +BUG_ON (atomic_read (&ioq->ref) <=0 ); +if (!atomic_dec_and_test (&ioq->ref)) +return ; +BUG_ON (ioq->nr_queued); +BUG_ON (ioq->entity.tree !=NULL ); +BUG_ON (elv_ioq_busy (ioq)); +BUG_ON (efqd->active_queue == ioq); + +/* Can be called by outgoing elevator. Don't use q */ +BUG_ON (!e->ops->elevator_free_sched_queue_fn); + + e->ops->elevator_free_sched_queue_fn (e, ioq->sched_queue); +elv_log_ioq (efqd, ioq,"put_queue" ); +elv_free_ioq (ioq); +} +EXPORT_SYMBOL (elv_put_ioq); + +void elv_release_ioq (struct elevator_queue *e,struct io_queue **ioq_ptr) +{ +struct io_queue *ioq = *ioq_ptr; + +if (ioq !=NULL ) { +/* Drop the reference taken by the io group */ +elv_put_ioq (ioq); + *ioq_ptr =NULL ; + } +} + +/* +* Normally next io queue to be served is selected from the service tree. +* This function allows one to choose a specific io queue to run next +* out of order. This is primarily to accomodate the close_cooperator +* feature of cfq. +* +* Currently it is done only for root level as to begin with supporting +* close cooperator feature only for root group to make sure default +* cfq behavior in flat hierarchy is not changed. +*/ +void elv_set_next_ioq (struct request_queue *q,struct io_queue *ioq) +{ +struct elv_fq_data *efqd = &q->elevator->efqd; +struct io_entity *entity = &ioq->entity; +struct io_sched_data *sd = &efqd->root_group->sched_data; +struct io_service_tree *st =io_entity_service_tree (entity); + +BUG_ON (efqd->active_queue !=NULL || sd->active_entity !=NULL ); +BUG_ON (!efqd->busy_queues); +BUG_ON (sd != entity->sched_data); +BUG_ON (!st); + +bfq_update_vtime (st); +bfq_active_extract (st, entity); + sd->active_entity = entity; + entity->service =0 ; +elv_log_ioq (efqd, ioq,"set_next_ioq" ); +} + +/* Get next queue for service. */ +struct io_queue *elv_get_next_ioq (struct request_queue *q,int extract) +{ +struct elv_fq_data *efqd = &q->elevator->efqd; +struct io_entity *entity =NULL ; +struct io_queue *ioq =NULL ; +struct io_sched_data *sd; + +/* +* We should not call lookup when an entity is active, as doing +* lookup can result in an erroneous vtime jump. +*/ +BUG_ON (efqd->active_queue !=NULL ); + +if (!efqd->busy_queues) +return NULL ; + + sd = &efqd->root_group->sched_data; + entity =bfq_lookup_next_entity (sd,1 ); +block/elevator-fq.c line 1145
Quoted 10 lines
block/elevator-fq.c new file +10
-0+BUG_ON (!entity); +if (extract) + entity->service =0 ; + ioq =io_entity_to_ioq (entity); + +return ioq; +} + +/* +* coop tells that io scheduler selected a queue for us and we did not coop?
Quoted 893 lines
+ * select the next queue based on fairness.
+ */block/elevator-fq.c new file +868
-0+static void __elv_set_active_ioq (struct elv_fq_data *efqd,struct io_queue *ioq, +int coop) +{ +struct request_queue *q = efqd->queue; + +if (ioq) { +elv_log_ioq (efqd, ioq,"set_active, busy=%d" , + efqd->busy_queues); + ioq->slice_end =0 ; + +elv_clear_ioq_wait_request (ioq); +elv_clear_ioq_must_dispatch (ioq); +elv_mark_ioq_slice_new (ioq); + +del_timer (&efqd->idle_slice_timer); + } + + efqd->active_queue = ioq; + +/* Let iosched know if it wants to take some action */ +if (ioq) { +if (q->elevator->ops->elevator_active_ioq_set_fn) + q->elevator->ops->elevator_active_ioq_set_fn (q, + ioq->sched_queue, coop); + } +} + +/* Get and set a new active queue for service. */ +struct io_queue *elv_set_active_ioq (struct request_queue *q, +struct io_queue *ioq) +{ +struct elv_fq_data *efqd = &q->elevator->efqd; +int coop =0 ; + +if (!ioq) + ioq =elv_get_next_ioq (q,1 ); +else { +elv_set_next_ioq (q, ioq); +/* +* io scheduler selected the next queue for us. Pass this +* this info back to io scheudler. cfq currently uses it +* to reset coop flag on the queue. +*/ + coop =1 ; + } +__elv_set_active_ioq (efqd, ioq, coop); +return ioq; +} + +void elv_reset_active_ioq (struct elv_fq_data *efqd) +{ +struct request_queue *q = efqd->queue; +struct io_queue *ioq =elv_active_ioq (efqd->queue->elevator); + +if (q->elevator->ops->elevator_active_ioq_reset_fn) + q->elevator->ops->elevator_active_ioq_reset_fn (q, + ioq->sched_queue); + efqd->active_queue =NULL ; +del_timer (&efqd->idle_slice_timer); +} + +void elv_activate_ioq (struct io_queue *ioq,int add_front) +{ +bfq_activate_entity (&ioq->entity, add_front); +} + +void elv_deactivate_ioq (struct elv_fq_data *efqd,struct io_queue *ioq, +int requeue) +{ +bfq_deactivate_entity (&ioq->entity, requeue); +} + +/* Called when an inactive queue receives a new request. */ +void elv_add_ioq_busy (struct elv_fq_data *efqd,struct io_queue *ioq) +{ +BUG_ON (elv_ioq_busy (ioq)); +BUG_ON (ioq == efqd->active_queue); +elv_log_ioq (efqd, ioq,"add to busy" ); +elv_activate_ioq (ioq,0 ); +elv_mark_ioq_busy (ioq); + efqd->busy_queues++; +if (elv_ioq_class_rt (ioq)) { +struct io_group *iog =ioq_to_io_group (ioq); + iog->busy_rt_queues++; + } +} + +void elv_del_ioq_busy (struct elevator_queue *e,struct io_queue *ioq, +int requeue) +{ +struct elv_fq_data *efqd = &e->efqd; + +BUG_ON (!elv_ioq_busy (ioq)); +BUG_ON (ioq->nr_queued); +elv_log_ioq (efqd, ioq,"del from busy" ); +elv_clear_ioq_busy (ioq); +BUG_ON (efqd->busy_queues ==0 ); + efqd->busy_queues--; +if (elv_ioq_class_rt (ioq)) { +struct io_group *iog =ioq_to_io_group (ioq); + iog->busy_rt_queues--; + } + +elv_deactivate_ioq (efqd, ioq, requeue); +} + +/* +* Do the accounting. Determine how much service (in terms of time slices) +* current queue used and adjust the start, finish time of queue and vtime +* of the tree accordingly. +* +* Determining the service used in terms of time is tricky in certain +* situations. Especially when underlying device supports command queuing +* and requests from multiple queues can be there at same time, then it +* is not clear which queue consumed how much of disk time. +* +* To mitigate this problem, cfq starts the time slice of the queue only +* after first request from the queue has completed. This does not work +* very well if we expire the queue before we wait for first and more +* request to finish from the queue. For seeky queues, we will expire the +* queue after dispatching few requests without waiting and start dispatching +* from next queue. +* +* Not sure how to determine the time consumed by queue in such scenarios. +* Currently as a crude approximation, we are charging 25% of time slice +* for such cases. A better mechanism is needed for accurate accounting. +*/ +void __elv_ioq_slice_expired (struct request_queue *q,struct io_queue *ioq) +{ +struct elv_fq_data *efqd = &q->elevator->efqd; +struct io_entity *entity = &ioq->entity; +long slice_unused =0 , slice_used =0 , slice_overshoot =0 ; + +assert_spin_locked (q->queue_lock); +elv_log_ioq (efqd, ioq,"slice expired" ); + +if (elv_ioq_wait_request (ioq)) +del_timer (&efqd->idle_slice_timer); + +elv_clear_ioq_wait_request (ioq); + +/* +* if ioq->slice_end = 0, that means a queue was expired before first +* reuqest from the queue got completed. Of course we are not planning +* to idle on the queue otherwise we would not have expired it. +* +* Charge for the 25% slice in such cases. This is not the best thing +* to do but at the same time not very sure what's the next best +* thing to do. +* +* This arises from that fact that we don't have the notion of +* one queue being operational at one time. io scheduler can dispatch +* requests from multiple queues in one dispatch round. Ideally for +* more accurate accounting of exact disk time used by disk, one +* should dispatch requests from only one queue and wait for all +* the requests to finish. But this will reduce throughput. +*/ +if (!ioq->slice_end) + slice_used = entity->budget/4 ; +else { +if (time_after (ioq->slice_end, jiffies)) { + slice_unused = ioq->slice_end - jiffies; +if (slice_unused == entity->budget) { +/* +* queue got expired immediately after +* completing first request. Charge 25% of +* slice. +*/ + slice_used = entity->budget/4 ; + }else + slice_used = entity->budget - slice_unused; + }else { + slice_overshoot = jiffies - ioq->slice_end; + slice_used = entity->budget + slice_overshoot; + } + } + +elv_log_ioq (efqd, ioq,"sl_end=%lx, jiffies=%lx" , ioq->slice_end, + jiffies); +elv_log_ioq (efqd, ioq,"sl_used=%ld, budget=%ld overshoot=%ld" , + slice_used, entity->budget, slice_overshoot); +elv_ioq_served (ioq, slice_used); + +BUG_ON (ioq != efqd->active_queue); +elv_reset_active_ioq (efqd); + +if (!ioq->nr_queued) +elv_del_ioq_busy (q->elevator, ioq,1 ); +else +elv_activate_ioq (ioq,0 ); +} +EXPORT_SYMBOL (__elv_ioq_slice_expired); + +/* +* Expire the ioq. +*/ +void elv_ioq_slice_expired (struct request_queue *q) +{ +struct io_queue *ioq =elv_active_ioq (q->elevator); + +if (ioq) +__elv_ioq_slice_expired (q, ioq); +} + +/* +* Check if new_cfqq should preempt the currently active queue. Return 0 for +* no or if we aren't sure, a 1 will cause a preemption attempt. +*/ +int elv_should_preempt (struct request_queue *q,struct io_queue *new_ioq, +struct request *rq) +{ +struct io_queue *ioq; +struct elevator_queue *eq = q->elevator; +struct io_entity *entity, *new_entity; + + ioq =elv_active_ioq (eq); + +if (!ioq) +return 0 ; + + entity = &ioq->entity; + new_entity = &new_ioq->entity; + +/* +* Allow an RT request to pre-empt an ongoing non-RT cfqq timeslice. +*/ + +if (new_entity->ioprio_class ==IOPRIO_CLASS_RT + && entity->ioprio_class !=IOPRIO_CLASS_RT ) +return 1 ; +/* +* Allow an BE request to pre-empt an ongoing IDLE clas timeslice. +*/ + +if (new_entity->ioprio_class ==IOPRIO_CLASS_BE + && entity->ioprio_class ==IOPRIO_CLASS_IDLE ) +return 1 ; + +/* +* Check with io scheduler if it has additional criterion based on +* which it wants to preempt existing queue. +*/ +if (eq->ops->elevator_should_preempt_fn) +return eq->ops->elevator_should_preempt_fn (q, +ioq_sched_queue (new_ioq), rq); + +return 0 ; +} + +static void elv_preempt_queue (struct request_queue *q,struct io_queue *ioq) +{ +elv_log_ioq (&q->elevator->efqd, ioq,"preempt" ); +elv_ioq_slice_expired (q); + +/* +* Put the new queue at the front of the of the current list, +* so we know that it will be selected next. +*/ + +elv_activate_ioq (ioq,1 ); +elv_ioq_set_slice_end (ioq,0 ); +elv_mark_ioq_slice_new (ioq); +} + +void elv_ioq_request_add (struct request_queue *q,struct request *rq) +{ +struct elv_fq_data *efqd = &q->elevator->efqd; +struct io_queue *ioq = rq->ioq; + +if (!elv_iosched_fair_queuing_enabled (q->elevator)) +return ; + +BUG_ON (!efqd); +BUG_ON (!ioq); + efqd->rq_queued++; + ioq->nr_queued++; + +if (!elv_ioq_busy (ioq)) +elv_add_ioq_busy (efqd, ioq); + +elv_ioq_update_io_thinktime (ioq); +elv_ioq_update_idle_window (q->elevator, ioq, rq); + +if (ioq ==elv_active_ioq (q->elevator)) { +/* +* Remember that we saw a request from this process, but +* don't start queuing just yet. Otherwise we risk seeing lots +* of tiny requests, because we disrupt the normal plugging +* and merging. If the request is already larger than a single +* page, let it rip immediately. For that case we assume that +* merging is already done. Ditto for a busy system that +* has other work pending, don't risk delaying until the +* idle timer unplug to continue working. +*/ +if (elv_ioq_wait_request (ioq)) { +if (blk_rq_bytes (rq) >PAGE_CACHE_SIZE || + efqd->busy_queues >1 ) { +del_timer (&efqd->idle_slice_timer); +blk_start_queueing (q); + } +elv_mark_ioq_must_dispatch (ioq); + } + }else if (elv_should_preempt (q, ioq, rq)) { +/* +* not the active queue - expire current slice if it is +* idle and has expired it's mean thinktime or this new queue +* has some old slice time left and is of higher priority or +* this new queue is RT and the current one is BE +*/ +elv_preempt_queue (q, ioq); +blk_start_queueing (q); + } +} + +void elv_idle_slice_timer (unsigned long data) +{ +struct elv_fq_data *efqd = (struct elv_fq_data *)data; +struct io_queue *ioq; +unsigned long flags; +struct request_queue *q = efqd->queue; + +elv_log (efqd,"idle timer fired" ); + +spin_lock_irqsave (q->queue_lock, flags); + + ioq = efqd->active_queue; + +if (ioq) { + +/* +* We saw a request before the queue expired, let it through +*/ +if (elv_ioq_must_dispatch (ioq)) + goto out_kick; + +/* +* expired +*/ +if (elv_ioq_slice_used (ioq)) + goto expire; + +/* +* only expire and reinvoke request handler, if there are +* other queues with pending requests +*/ +if (!elv_nr_busy_ioq (q->elevator)) + goto out_cont; + +/* +* not expired and it has a request pending, let it dispatch +*/ +if (ioq->nr_queued) + goto out_kick; + } +expire: +elv_ioq_slice_expired (q); +out_kick: +elv_schedule_dispatch (q); +out_cont: +spin_unlock_irqrestore (q->queue_lock, flags); +} + +void elv_ioq_arm_slice_timer (struct request_queue *q) +{ +struct elv_fq_data *efqd = &q->elevator->efqd; +struct io_queue *ioq =elv_active_ioq (q->elevator); +unsigned long sl; + +BUG_ON (!ioq); + +/* +* SSD device without seek penalty, disable idling. But only do so +* for devices that support queuing, otherwise we still have a problem +* with sync vs async workloads. +*/ +if (blk_queue_nonrot (q) && efqd->hw_tag) +return ; + +/* +* still requests with the driver, don't idle +*/ +if (efqd->rq_in_driver) +return ; + +/* +* idle is disabled, either manually or by past process history +*/ +if (!efqd->elv_slice_idle || !elv_ioq_idle_window (ioq)) +return ; + +/* +* may be iosched got its own idling logic. In that case io +* schduler will take care of arming the timer, if need be. +*/ +if (q->elevator->ops->elevator_arm_slice_timer_fn) { + q->elevator->ops->elevator_arm_slice_timer_fn (q, + ioq->sched_queue); + }else { +elv_mark_ioq_wait_request (ioq); + sl = efqd->elv_slice_idle; +mod_timer (&efqd->idle_slice_timer, jiffies + sl); +elv_log_ioq (efqd, ioq,"arm idle: %lu" , sl); + } +} + +/* Common layer function to select the next queue to dispatch from */ +void *elv_fq_select_ioq (struct request_queue *q,int force) +{ +struct elv_fq_data *efqd = &q->elevator->efqd; +struct io_queue *new_ioq =NULL , *ioq =elv_active_ioq (q->elevator); +struct io_group *iog; + +if (!elv_nr_busy_ioq (q->elevator)) +return NULL ; + +if (ioq ==NULL ) + goto new_queue; + +/* +* Force dispatch. Continue to dispatch from current queue as long +* as it has requests. +*/ +if (unlikely (force)) { +if (ioq->nr_queued) + goto keep_queue; +else + goto expire; + } + +/* +* The active queue has run out of time, expire it and select new. +*/ +if (elv_ioq_slice_used (ioq) && !elv_ioq_must_dispatch (ioq)) + goto expire; + +/* +* If we have a RT cfqq waiting, then we pre-empt the current non-rt +* cfqq. +*/ + iog =ioq_to_io_group (ioq); + +if (!elv_ioq_class_rt (ioq) && iog->busy_rt_queues) { +/* +* We simulate this as cfqq timed out so that it gets to bank +* the remaining of its time slice. +*/ +elv_log_ioq (efqd, ioq,"preempt" ); + goto expire; + } + +/* +* The active queue has requests and isn't expired, allow it to +* dispatch. +*/ + +if (ioq->nr_queued) + goto keep_queue; + +/* +* If another queue has a request waiting within our mean seek +* distance, let it run. The expire code will check for close +* cooperators and put the close queue at the front of the service +* tree. +*/ + new_ioq =elv_close_cooperator (q, ioq,0 ); +if (new_ioq) + goto expire; + +/* +* No requests pending. If the active queue still has requests in +* flight or is idling for a new request, allow either of these +* conditions to happen (or time out) before selecting a new queue. +*/ + +if (timer_pending (&efqd->idle_slice_timer) || + (elv_ioq_nr_dispatched (ioq) &&elv_ioq_idle_window (ioq))) { + ioq =NULL ; + goto keep_queue; + } + +expire: +elv_ioq_slice_expired (q); +new_queue: + ioq =elv_set_active_ioq (q, new_ioq); +keep_queue: +return ioq; +} + +/* A request got removed from io_queue. Do the accounting */ +void elv_ioq_request_removed (struct elevator_queue *e,struct request *rq) +{ +struct io_queue *ioq; +struct elv_fq_data *efqd; + +if (!elv_iosched_fair_queuing_enabled (e)) +return ; + + ioq = rq->ioq; +BUG_ON (!ioq); + ioq->nr_queued--; + + efqd = ioq->efqd; +BUG_ON (!efqd); + efqd->rq_queued--; + +if (elv_ioq_busy (ioq) && (elv_active_ioq (e) != ioq) && !ioq->nr_queued) +elv_del_ioq_busy (e, ioq,1 ); +} + +/* A request got dispatched. Do the accounting. */ +void elv_fq_dispatched_request (struct elevator_queue *e,struct request *rq) +{ +struct io_queue *ioq = rq->ioq; + +if (!elv_iosched_fair_queuing_enabled (e)) +return ; + +BUG_ON (!ioq); +elv_ioq_request_dispatched (ioq); +elv_ioq_request_removed (e, rq); +elv_clear_ioq_must_dispatch (ioq); +} + +void elv_fq_activate_rq (struct request_queue *q,struct request *rq) +{ +struct elv_fq_data *efqd = &q->elevator->efqd; + +if (!elv_iosched_fair_queuing_enabled (q->elevator)) +return ; + + efqd->rq_in_driver++; +elv_log_ioq (efqd,rq_ioq (rq),"activate rq, drv=%d" , + efqd->rq_in_driver); +} + +void elv_fq_deactivate_rq (struct request_queue *q,struct request *rq) +{ +struct elv_fq_data *efqd = &q->elevator->efqd; + +if (!elv_iosched_fair_queuing_enabled (q->elevator)) +return ; + +WARN_ON (!efqd->rq_in_driver); + efqd->rq_in_driver--; +elv_log_ioq (efqd,rq_ioq (rq),"deactivate rq, drv=%d" , + efqd->rq_in_driver); +} + +/* +* Update hw_tag based on peak queue depth over 50 samples under +* sufficient load. +*/ +static void elv_update_hw_tag (struct elv_fq_data *efqd) +{ +if (efqd->rq_in_driver > efqd->rq_in_driver_peak) + efqd->rq_in_driver_peak = efqd->rq_in_driver; + +if (efqd->rq_queued <=ELV_HW_QUEUE_MIN && + efqd->rq_in_driver <=ELV_HW_QUEUE_MIN ) +return ; + +if (efqd->hw_tag_samples++ <50 ) +return ; + +if (efqd->rq_in_driver_peak >=ELV_HW_QUEUE_MIN ) + efqd->hw_tag =1 ; +else + efqd->hw_tag =0 ; + + efqd->hw_tag_samples =0 ; + efqd->rq_in_driver_peak =0 ; +} + +/* +* If ioscheduler has functionality of keeping track of close cooperator, check +* with it if it has got a closely co-operating queue. +*/ +static inline struct io_queue *elv_close_cooperator (struct request_queue *q, +struct io_queue *ioq,int probe) +{ +struct elevator_queue *e = q->elevator; +struct io_queue *new_ioq =NULL ; + +/* +* Currently this feature is supported only for flat hierarchy or +* root group queues so that default cfq behavior is not changed. +*/ +if (!is_root_group_ioq (q, ioq)) +return NULL ; + +if (q->elevator->ops->elevator_close_cooperator_fn) + new_ioq = e->ops->elevator_close_cooperator_fn (q, + ioq->sched_queue, probe); + +/* Only select co-operating queue if it belongs to root group */ +if (new_ioq && !is_root_group_ioq (q, new_ioq)) +return NULL ; + +return new_ioq; +} + +/* A request got completed from io_queue. Do the accounting. */ +void elv_ioq_completed_request (struct request_queue *q,struct request *rq) +{ +const int sync =rq_is_sync (rq); +struct io_queue *ioq; +struct elv_fq_data *efqd = &q->elevator->efqd; + +if (!elv_iosched_fair_queuing_enabled (q->elevator)) +return ; + + ioq = rq->ioq; + +elv_log_ioq (efqd, ioq,"complete" ); + +elv_update_hw_tag (efqd); + +WARN_ON (!efqd->rq_in_driver); +WARN_ON (!ioq->dispatched); + efqd->rq_in_driver--; + ioq->dispatched--; + +if (sync) + ioq->last_end_request = jiffies; + +/* +* If this is the active queue, check if it needs to be expired, +* or if we want to idle in case it has no pending requests. +*/ + +if (elv_active_ioq (q->elevator) == ioq) { +if (elv_ioq_slice_new (ioq)) { +elv_ioq_set_prio_slice (q, ioq); +elv_clear_ioq_slice_new (ioq); + } +/* +* If there are no requests waiting in this queue, and +* there are other queues ready to issue requests, AND +* those other queues are issuing requests within our +* mean seek distance, give them a chance to run instead +* of idling. +*/ +if (elv_ioq_slice_used (ioq) ||elv_ioq_class_idle (ioq)) +elv_ioq_slice_expired (q); +else if (!ioq->nr_queued && !elv_close_cooperator (q, ioq,1 ) + && sync && !rq_noidle (rq)) +elv_ioq_arm_slice_timer (q); + } + +if (!efqd->rq_in_driver) +elv_schedule_dispatch (q); +} + +struct io_group *io_lookup_io_group_current (struct request_queue *q) +{ +struct elv_fq_data *efqd = &q->elevator->efqd; + +return efqd->root_group; +} +EXPORT_SYMBOL (io_lookup_io_group_current); + +void *io_group_async_queue_prio (struct io_group *iog,int ioprio_class, +int ioprio) +{ +struct io_queue *ioq =NULL ; + +switch (ioprio_class) { +case IOPRIO_CLASS_RT : + ioq = iog->async_queue[0 ][ioprio]; +break ; +case IOPRIO_CLASS_BE : + ioq = iog->async_queue[1 ][ioprio]; +break ; +case IOPRIO_CLASS_IDLE : + ioq = iog->async_idle_queue; +break ; +default : +BUG (); + } + +if (ioq) +return ioq->sched_queue; +return NULL ; +} +EXPORT_SYMBOL (io_group_async_queue_prio); + +void io_group_set_async_queue (struct io_group *iog,int ioprio_class, +int ioprio,struct io_queue *ioq) +{ +switch (ioprio_class) { +case IOPRIO_CLASS_RT : + iog->async_queue[0 ][ioprio] = ioq; +break ; +case IOPRIO_CLASS_BE : + iog->async_queue[1 ][ioprio] = ioq; +break ; +case IOPRIO_CLASS_IDLE : + iog->async_idle_queue = ioq; +break ; +default : +BUG (); + } + +/* +* Take the group reference and pin the queue. Group exit will +* clean it up +*/ +elv_get_ioq (ioq); +} +EXPORT_SYMBOL (io_group_set_async_queue); + +/* +* Release all the io group references to its async queues. +*/ +void io_put_io_group_queues (struct elevator_queue *e,struct io_group *iog) +{ +int i, j; + +for (i =0 ; i <2 ; i++) +for (j =0 ; j <IOPRIO_BE_NR ; j++) +elv_release_ioq (e, &iog->async_queue[i][j]); + +/* Free up async idle queue */ +elv_release_ioq (e, &iog->async_idle_queue); +} + +struct io_group *io_alloc_root_group (struct request_queue *q, +struct elevator_queue *e,void *key) +{ +struct io_group *iog; +int i; + + iog =kmalloc_node (sizeof (*iog),GFP_KERNEL | __GFP_ZERO, q->node); +if (iog ==NULL ) +return NULL ; + +for (i =0 ; i <IO_IOPRIO_CLASSES ; i++) + iog->sched_data.service_tree[i] =IO_SERVICE_TREE_INIT ; + +return iog; +} + +void io_free_root_group (struct elevator_queue *e) +{ +struct io_group *iog = e->efqd.root_group; +struct io_service_tree *st; +int i; + +for (i =0 ; i <IO_IOPRIO_CLASSES ; i++) { + st = iog->sched_data.service_tree + i; +io_flush_idle_tree (st); + } + +io_put_io_group_queues (e, iog); +kfree (iog); +} + +static void elv_slab_kill (void ) +{ +/* +* Caller already ensured that pending RCU callbacks are completed, +* so we should have no busy allocations at this point. +*/ +if (elv_ioq_pool) +kmem_cache_destroy (elv_ioq_pool); +} + +static int __init elv_slab_setup (void ) +{ + elv_ioq_pool =KMEM_CACHE (io_queue,0 ); +if (!elv_ioq_pool) + goto fail; + +return 0 ; +fail: +elv_slab_kill (); +return -ENOMEM ; +} + +/* Initialize fair queueing data associated with elevator */ +int elv_init_fq_data (struct request_queue *q,struct elevator_queue *e) +{ +struct io_group *iog; +struct elv_fq_data *efqd = &e->efqd; + +if (!elv_iosched_fair_queuing_enabled (e)) +return 0 ; + + iog =io_alloc_root_group (q, e, efqd); +if (iog ==NULL ) +return 1 ; + + efqd->root_group = iog; + efqd->queue = q; + +init_timer (&efqd->idle_slice_timer); + efqd->idle_slice_timer.function = elv_idle_slice_timer; + efqd->idle_slice_timer.data = (unsigned long ) efqd; + +INIT_WORK (&efqd->unplug_work, elv_kick_queue); + + efqd->elv_slice[0 ] = elv_slice_async; + efqd->elv_slice[1 ] = elv_slice_sync; + efqd->elv_slice_idle = elv_slice_idle; + efqd->hw_tag =1 ; + +return 0 ; +} + +/* +* elv_exit_fq_data is called before we call elevator_exit_fn. Before +* we ask elevator to cleanup its queues, we do the cleanup here so +* that all the group and idle tree references to ioq are dropped. Later +* during elevator cleanup, ioc reference will be dropped which will lead +* to removal of ioscheduler queue as well as associated ioq object. +*/ +void elv_exit_fq_data (struct elevator_queue *e) +{ +struct elv_fq_data *efqd = &e->efqd; + +if (!elv_iosched_fair_queuing_enabled (e)) +return ; + +elv_shutdown_timer_wq (e); + +BUG_ON (timer_pending (&efqd->idle_slice_timer)); +io_free_root_group (e); +} + +/* +* This is called after the io scheduler has cleaned up its data structres. +* I don't think that this function is required. Right now just keeping it +* because cfq cleans up timer and work queue again after freeing up +* io contexts. To me io scheduler has already been drained out, and all +* the active queue have already been expired so time and work queue should +* not been activated during cleanup process. +* +* Keeping it here for the time being. Will get rid of it later. +*/ +void elv_exit_fq_data_post (struct elevator_queue *e) +{ +struct elv_fq_data *efqd = &e->efqd; + +if (!elv_iosched_fair_queuing_enabled (e)) +return ; + +elv_shutdown_timer_wq (e); +BUG_ON (timer_pending (&efqd->idle_slice_timer)); +} + + +static int __init elv_fq_init (void ) +{ +if (elv_slab_setup ()) +return -ENOMEM ; + +/* could be 0 on HZ < 1000 setups */ + +if (!elv_slice_async) + elv_slice_async =1 ; + +if (!elv_slice_idle) + elv_slice_idle =1 ; + +return 0 ; +} + +module_init (elv_fq_init);block/elevator-fq.h new file +17
-0@@ -0,0 +1,473 @@ +/* +* BFQ: data structures and common functions prototypes. +* +* Based on ideas and code from CFQ: +* Copyright (C) 2003 Jens Axboe <[email protected]> +* +* Copyright (C) 2008 Fabio Checconi <[email protected]> +* Paolo Valente <[email protected]> +* Copyright (C) 2009 Vivek Goyal <[email protected]> +* Nauman Rafique <[email protected]> +*/ + +#include <linux/blkdev.h> + +#ifndef _BFQ_SCHED_H +#define _BFQ_SCHED_H +
Quoted 9 lines
block/elevator-fq.h line 22
Quoted 5 lines
block/elevator-fq.h new file +5
-0+#define IO_IOPRIO_CLASSES 3 + +typedef u64 bfq_timestamp_t ; +typedef unsigned long bfq_weight_t ; +typedef unsigned long bfq_service_t ;Does this abstraction really provide any benefit? Why not directly use the standard C types, make the code easier to read.
I have no strong opinions on that, during debugging it helped a lot to identify the role of variables in the code, but common practice in the kernel is avoiding typedefs, so they can go now.
Quoted 66 lines
block/elevator-fq.h line 32
Quoted 10 lines
block/elevator-fq.h new file +10
-0+struct io_entity ; +struct io_queue ; + +#ifdef CONFIG_ELV_FAIR_QUEUING + +#define ELV_ATTR (name) \ + __ATTR(name, S_IRUGO|S_IWUSR, elv_##name##_show, elv_##name##_store) + +/** +* struct bfq_service_tree - per ioprio_class service tree. Comment is old, does not reflect the newer name
Quoted 15 lines
block/elevator-fq.h new file +15
-0+* @active: tree for active entities (i.e., those backlogged). +* @idle: tree for idle entities (i.e., those not backlogged, with V <= F_i). +* @first_idle: idle entity with minimum F_i. +* @last_idle: idle entity with maximum F_i. +* @vtime: scheduler virtual time. +* @wsum: scheduler weight sum; active and idle entities contribute to it. +* +* Each service tree represents a B-WF2Q+ scheduler on its own. Each +* ioprio_class has its own independent scheduler, and so its own +* bfq_service_tree. All the fields are protected by the queue lock +* of the containing efqd. +*/ +struct io_service_tree { +struct rb_root active; +struct rb_root idle;block/elevator-fq.h line 57
Quoted 10 lines
block/elevator-fq.h new file +10
-0+ +struct io_entity *first_idle; +struct io_entity *last_idle; + +bfq_timestamp_t vtime; +bfq_weight_t wsum; +}; + +/** +* struct bfq_sched_data - multi-class scheduler. Again the naming convention is broken, you need to change several bfq's to io's :)
Quoted 24 lines
block/elevator-fq.h new file +24
-0+* @active_entity: entity under service. +* @next_active: head-of-the-line entity in the scheduler. +* @service_tree: array of service trees, one per ioprio_class. +* +* bfq_sched_data is the basic scheduler queue. It supports three +* ioprio_classes, and can be used either as a toplevel queue or as +* an intermediate queue on a hierarchical setup. +* @next_active points to the active entity of the sched_data service +* trees that will be scheduled next. +* +* The supported ioprio_classes are the same as in CFQ, in descending +* priority order, IOPRIO_CLASS_RT, IOPRIO_CLASS_BE, IOPRIO_CLASS_IDLE. +* Requests from higher priority queues are served before all the +* requests from lower priority queues; among requests of the same +* queue requests are served according to B-WF2Q+. +* All the fields are protected by the queue lock of the containing bfqd. +*/ +struct io_sched_data { +struct io_entity *active_entity; +struct io_service_tree service_tree[IO_IOPRIO_CLASSES ]; +}; + +/** +* struct bfq_entity - schedulable entity.
Quoted 9 lines
block/elevator-fq.h line 86
Quoted 5 lines
block/elevator-fq.h new file +5
-0+* @rb_node: service_tree member. +* @on_st: flag, true if the entity is on a tree (either the active or +* the idle one of its service_tree). +* @finish: B-WF2Q+ finish timestamp (aka F_i). +* @start: B-WF2Q+ start timestamp (aka S_i). Could you mention what key is used in the
rb_tree? start, finish sounds like a range, so my suspicion is that start is used.
finish is used as the key, and min_start keeps the minimum ->start for the subtree rooted at the given entity (as said in the comment below).
Quoted 7 lines
block/elevator-fq.h line 89
Quoted 3 lines
block/elevator-fq.h new file +3
-0+* @tree: tree the entity is enqueued into; %NULL if not on a tree. +* @min_start: minimum start time of the (active) subtree rooted at +* this entity; used for O(log N) lookups into active trees. Used for O(log N) makes no sense to me, RBTree has a worst case lookup time of O(log N), but what is the comment saying?
it's badly written (my fault), but it intended to say that this field is used to allow the lookups to be done in O(log N). without augmenting the RB tree with min_start, lookups could not be done in O(log N), because we want a constrained minimum search.
Quoted 64 lines
Quoted 64 lines
block/elevator-fq.h new file +64
-0+* @service: service received during the last round of service. +* @budget: budget used to calculate F_i; F_i = S_i + @budget / @weight. +* @weight: weight of the queue, calculated as IOPRIO_BE_NR - @ioprio. +* @parent: parent entity, for hierarchical scheduling. +* @my_sched_data: for non-leaf nodes in the cgroup hierarchy, the +* associated scheduler queue, %NULL on leaf nodes. +* @sched_data: the scheduler queue this entity belongs to. +* @ioprio: the ioprio in use. +* @new_ioprio: when an ioprio change is requested, the new ioprio value +* @ioprio_class: the ioprio_class in use. +* @new_ioprio_class: when an ioprio_class change is requested, the new +* ioprio_class value. +* @ioprio_changed: flag, true when the user requested an ioprio or +* ioprio_class change. +* +* A bfq_entity is used to represent either a bfq_queue (leaf node in the +* cgroup hierarchy) or a bfq_group into the upper level scheduler. Each +* entity belongs to the sched_data of the parent group in the cgroup +* hierarchy. Non-leaf entities have also their own sched_data, stored +* in @my_sched_data. +* +* Each entity stores independently its priority values; this would allow +* different weights on different devices, but this functionality is not +* exported to userspace by now. Priorities are updated lazily, first +* storing the new values into the new_* fields, then setting the +* @ioprio_changed flag. As soon as there is a transition in the entity +* state that allows the priority update to take place the effective and +* the requested priority values are synchronized. +* +* The weight value is calculated from the ioprio to export the same +* interface as CFQ. When dealing with ``well-behaved'' queues (i.e., +* queues that do not spend too much time to consume their budget and +* have true sequential behavior, and when there are no external factors +* breaking anticipation) the relative weights at each level of the +* cgroups hierarchy should be guaranteed. +* All the fields are protected by the queue lock of the containing bfqd. +*/ +struct io_entity { +struct rb_node rb_node; + +int on_st; + +bfq_timestamp_t finish; +bfq_timestamp_t start; + +struct rb_root *tree; + +bfq_timestamp_t min_start; + +bfq_service_t service, budget; +bfq_weight_t weight; + +struct io_entity *parent; + +struct io_sched_data *my_sched_data; +struct io_sched_data *sched_data; + +unsigned short ioprio, new_ioprio; +unsigned short ioprio_class, new_ioprio_class; + +int ioprio_changed; +}; + +/*
Quoted 9 lines
block/elevator-fq.h line 158
Quoted 5 lines
block/elevator-fq.h new file +5
-0+* A common structure embedded by every io scheduler into their respective +* queue structure. +*/ +struct io_queue { +struct io_entity entity;So the
io_queuehas an abstract entity calledio_entitythat contains it QoS parameters? Correct?
yes
Quoted 9 lines
block/elevator-fq.h line 164
Quoted 6 lines
block/elevator-fq.h new file +6
-0+atomic_t ref; +unsigned int flags; + +/* Pointer to generic elevator data structure */ +struct elv_fq_data *efqd; +pid_t pid;Why do we store the pid?
originally it was for logging purposes
Quoted 570 lines
Quoted 14 lines
block/elevator-fq.h new file +14
-0+ +/* Number of requests queued on this io queue */ +unsigned long nr_queued; + +/* Requests dispatched from this queue */ +int dispatched; + +/* Keep a track of think time of processes in this queue */ +unsigned long last_end_request; +unsigned long ttime_total; +unsigned long ttime_samples; +unsigned long ttime_mean; + +unsigned long slice_end;block/elevator-fq.h line 188
Quoted 10 lines
block/elevator-fq.h new file +10
-0+ +/* Pointer to io scheduler's queue */ +void *sched_queue; +}; + +struct io_group { +struct io_sched_data sched_data; + +/* async_queue and idle_queue are used only for cfq */ +struct io_queue *async_queue[2 ][IOPRIO_BE_NR ];Again the 2 is confusing
block/elevator-fq.h line 198
Quoted 10 lines
block/elevator-fq.h new file +10
-0+struct io_queue *async_idle_queue; + +/* +* Used to track any pending rt requests so we can pre-empt current +* non-RT cfqq in service when this value is non-zero. +*/ +unsigned int busy_rt_queues; +}; + +struct elv_fq_data {What does fq stand for?
Quoted 5 lines
block/elevator-fq.h new file +5
-0+struct io_group *root_group; + +struct request_queue *queue; +unsigned int busy_queues; +block/elevator-fq.h line 213
Quoted 10 lines
block/elevator-fq.h new file +10
-0+/* Number of requests queued */ +int rq_queued; + +/* Pointer to the ioscheduler queue being served */ +void *active_queue; + +int rq_in_driver; +int hw_tag; +int hw_tag_samples; +int rq_in_driver_peak;Some comments of
_in_driverand_in_driver_peakwould be nice.Quoted 4 lines
block/elevator-fq.h new file +4
-0+ +/* +* elevator fair queuing layer has the capability to provide idling +* for ensuring fairness for processes doing dependent reads. block/elevator-fq.h line 227
Quoted 10 lines
block/elevator-fq.h new file +10
-0+* This might be needed to ensure fairness among two processes doing +* synchronous reads in two different cgroups. noop and deadline don't +* have any notion of anticipation/idling. As of now, these are the +* users of this functionality. +*/ +unsigned int elv_slice_idle; +struct timer_list idle_slice_timer; +struct work_struct unplug_work; + +unsigned int elv_slice[2 ];Why [2] makes the code hearder to read
Quoted 492 lines
block/elevator-fq.h new file +246
-0+}; + +extern int elv_slice_idle; +extern int elv_slice_async; + +/* Logging facilities. */ +#define elv_log_ioq(efqd, ioq, fmt, args...) \ + blk_add_trace_msg((efqd)->queue, "elv%d%c " fmt, (ioq)->pid, \ + elv_ioq_sync(ioq) ? 'S' : 'A', ##args) + +#define elv_log(efqd, fmt, args...) \ + blk_add_trace_msg((efqd)->queue, "elv " fmt, ##args) + +#define ioq_sample_valid (samples) ((samples) > 80) + +/* Some shared queue flag manipulation functions among elevators */ + +enum elv_queue_state_flags { + ELV_QUEUE_FLAG_busy =0 ,/* has requests or is under service */ + ELV_QUEUE_FLAG_sync,/* synchronous queue */ + ELV_QUEUE_FLAG_idle_window,/* elevator slice idling enabled */ + ELV_QUEUE_FLAG_wait_request,/* waiting for a request */ + ELV_QUEUE_FLAG_must_dispatch,/* must be allowed a dispatch */ + ELV_QUEUE_FLAG_slice_new,/* no requests dispatched in slice */ +ELV_QUEUE_FLAG_NR , +}; + +#define ELV_IO_QUEUE_FLAG_FNS (name) \ +static inline void elv_mark_ioq_##name(struct io_queue *ioq) \ +{ \ + (ioq)->flags |= (1 << ELV_QUEUE_FLAG_##name); \ +} \ +static inline void elv_clear_ioq_##name(struct io_queue *ioq) \ +{ \ + (ioq)->flags &= ~(1 << ELV_QUEUE_FLAG_##name); \ +} \ +static inline int elv_ioq_##name(struct io_queue *ioq) \ +{ \ + return ((ioq)->flags & (1 << ELV_QUEUE_FLAG_##name)) != 0; \ +} + +ELV_IO_QUEUE_FLAG_FNS (busy) +ELV_IO_QUEUE_FLAG_FNS (sync) +ELV_IO_QUEUE_FLAG_FNS (wait_request) +ELV_IO_QUEUE_FLAG_FNS(must_dispatch) +ELV_IO_QUEUE_FLAG_FNS(idle_window) +ELV_IO_QUEUE_FLAG_FNS(slice_new) + +static inline struct io_service_tree * +io_entity_service_tree(struct io_entity *entity) +{ + struct io_sched_data *sched_data = entity->sched_data; +unsigned int idx = entity->ioprio_class -1 ; + + BUG_ON(idx >= IO_IOPRIO_CLASSES); + BUG_ON(sched_data == NULL); + + return sched_data->service_tree + idx; +} + +/* A request got dispatched from the io_queue. Do the accounting. */ +static inline void elv_ioq_request_dispatched(struct io_queue *ioq) +{ + ioq->dispatched++; +} + +static inline int elv_ioq_slice_used (struct io_queue *ioq) +{ +if (elv_ioq_slice_new (ioq)) +return 0 ; +if (time_before (jiffies, ioq->slice_end)) +return 0 ; + +return 1 ; +} + +/* How many request are currently dispatched from the queue */ +static inline int elv_ioq_nr_dispatched (struct io_queue *ioq) +{ +return ioq->dispatched; +} + +/* How many request are currently queued in the queue */ +static inline int elv_ioq_nr_queued (struct io_queue *ioq) +{ +return ioq->nr_queued; +} + +static inline void elv_get_ioq (struct io_queue *ioq) +{ +atomic_inc (&ioq->ref); +} + +static inline void elv_ioq_set_slice_end (struct io_queue *ioq, +unsigned long slice_end) +{ + ioq->slice_end = slice_end; +} + +static inline int elv_ioq_class_idle (struct io_queue *ioq) +{ +return ioq->entity.ioprio_class ==IOPRIO_CLASS_IDLE ; +} + +static inline int elv_ioq_class_rt (struct io_queue *ioq) +{ +return ioq->entity.ioprio_class ==IOPRIO_CLASS_RT ; +} + +static inline int elv_ioq_ioprio_class (struct io_queue *ioq) +{ +return ioq->entity.new_ioprio_class; +} + +static inline int elv_ioq_ioprio (struct io_queue *ioq) +{ +return ioq->entity.new_ioprio; +} + +static inline void elv_ioq_set_ioprio_class (struct io_queue *ioq, +int ioprio_class) +{ + ioq->entity.new_ioprio_class = ioprio_class; + ioq->entity.ioprio_changed =1 ; +} + +static inline void elv_ioq_set_ioprio (struct io_queue *ioq,int ioprio) +{ + ioq->entity.new_ioprio = ioprio; + ioq->entity.ioprio_changed =1 ; +} + +static inline void *ioq_sched_queue (struct io_queue *ioq) +{ +if (ioq) +return ioq->sched_queue; +return NULL ; +} + +static inline struct io_group *ioq_to_io_group (struct io_queue *ioq) +{ + return container_of(ioq->entity.sched_data, struct io_group, + sched_data); +} + +extern ssize_t elv_slice_idle_show (struct elevator_queue *q,char *name); +extern ssize_t elv_slice_idle_store (struct elevator_queue *q,const char *name, +size_t count); +extern ssize_t elv_slice_sync_show (struct elevator_queue *q,char *name); +extern ssize_t elv_slice_sync_store (struct elevator_queue *q,const char *name, +size_t count); +extern ssize_t elv_slice_async_show (struct elevator_queue *q,char *name); +extern ssize_t elv_slice_async_store (struct elevator_queue *q,const char *name, +size_t count); + +/* Functions used by elevator.c */ +extern int elv_init_fq_data (struct request_queue *q,struct elevator_queue *e); +extern void elv_exit_fq_data (struct elevator_queue *e); +extern void elv_exit_fq_data_post (struct elevator_queue *e); + +extern void elv_ioq_request_add (struct request_queue *q,struct request *rq); +extern void elv_ioq_request_removed (struct elevator_queue *e, +struct request *rq); +extern void elv_fq_dispatched_request (struct elevator_queue *e, +struct request *rq); + +extern void elv_fq_activate_rq (struct request_queue *q,struct request *rq); +extern void elv_fq_deactivate_rq (struct request_queue *q,struct request *rq); + +extern void elv_ioq_completed_request (struct request_queue *q, +struct request *rq); + +extern void *elv_fq_select_ioq (struct request_queue *q,int force); +extern struct io_queue *rq_ioq (struct request *rq); + +/* Functions used by io schedulers */ +extern void elv_put_ioq (struct io_queue *ioq); +extern void __elv_ioq_slice_expired (struct request_queue *q, +struct io_queue *ioq); +extern int elv_init_ioq (struct elevator_queue *eq,struct io_queue *ioq, +void *sched_queue,int ioprio_class,int ioprio,int is_sync); +extern void elv_schedule_dispatch (struct request_queue *q); +extern int elv_hw_tag (struct elevator_queue *e); +extern void *elv_active_sched_queue (struct elevator_queue *e); +extern int elv_mod_idle_slice_timer (struct elevator_queue *eq, +unsigned long expires); +extern int elv_del_idle_slice_timer (struct elevator_queue *eq); +extern unsigned int elv_get_slice_idle (struct elevator_queue *eq); +extern void *io_group_async_queue_prio (struct io_group *iog,int ioprio_class, +int ioprio); +extern void io_group_set_async_queue (struct io_group *iog,int ioprio_class, +int ioprio,struct io_queue *ioq); +extern struct io_group *io_lookup_io_group_current (struct request_queue *q); +extern int elv_nr_busy_ioq (struct elevator_queue *e); +extern struct io_queue *elv_alloc_ioq (struct request_queue *q,gfp_t gfp_mask); +extern void elv_free_ioq (struct io_queue *ioq); + +#else /* CONFIG_ELV_FAIR_QUEUING */ + +static inline int elv_init_fq_data (struct request_queue *q, +struct elevator_queue *e) +{ +return 0 ; +} + +static inline void elv_exit_fq_data (struct elevator_queue *e) {} +static inline void elv_exit_fq_data_post (struct elevator_queue *e) {} + +static inline void elv_fq_activate_rq (struct request_queue *q, +struct request *rq) +{ +} + +static inline void elv_fq_deactivate_rq (struct request_queue *q, +struct request *rq) +{ +} + +static inline void elv_fq_dispatched_request (struct elevator_queue *e, +struct request *rq) +{ +} + +static inline void elv_ioq_request_removed (struct elevator_queue *e, +struct request *rq) +{ +} + +static inline void elv_ioq_request_add (struct request_queue *q, +struct request *rq) +{ +} + +static inline void elv_ioq_completed_request (struct request_queue *q, +struct request *rq) +{ +} + +static inline void *ioq_sched_queue (struct io_queue *ioq) {return NULL ; } +static inline struct io_queue *rq_ioq (struct request *rq) {return NULL ; } +static inline void *elv_fq_select_ioq (struct request_queue *q,int force) +{ +return NULL ; +} +#endif /* CONFIG_ELV_FAIR_QUEUING */ +#endif /* _BFQ_SCHED_H */ block/elevator.c +35
-11@@ -231,6 +231,9 @@ static struct elevator_queue *elevator_alloc(struct request_queue *q,for (i =0 ; i <ELV_HASH_ENTRIES ; i++)INIT_HLIST_HEAD (&eq->hash[i]); +if (elv_init_fq_data (q, eq)) + goto err; +return eq; err:kfree (eq);@@ -301,9 +304,11 @@ EXPORT_SYMBOL(elevator_init);void elevator_exit (struct elevator_queue *e) {mutex_lock (&e->sysfs_lock); +elv_exit_fq_data (e);if (e->ops->elevator_exit_fn) e->ops->elevator_exit_fn (e); e->ops =NULL ; +elv_exit_fq_data_post (e);mutex_unlock (&e->sysfs_lock);kobject_put (&e->kobj);@@ -314,6 +319,8 @@ static void elv_activate_rq(struct request_queue *q, struct request *rq) {struct elevator_queue *e = q->elevator; +elv_fq_activate_rq (q, rq); +if (e->ops->elevator_activate_req_fn) e->ops->elevator_activate_req_fn (q, rq); }@@ -322,6 +329,8 @@ static void elv_deactivate_rq(struct request_queue *q, struct request *rq) {struct elevator_queue *e = q->elevator; +elv_fq_deactivate_rq (q, rq); +if (e->ops->elevator_deactivate_req_fn) e->ops->elevator_deactivate_req_fn (q, rq); }@@ -446,6 +455,7 @@ void elv_dispatch_sort(struct request_queue *q, struct request *rq)elv_rqhash_del (q, rq); q->nr_sorted--; +elv_fq_dispatched_request (q->elevator, rq); boundary = q->end_sector; stop_flags =REQ_SOFTBARRIER |REQ_HARDBARRIER |REQ_STARTED ;@@ -486,6 +496,7 @@ void elv_dispatch_add_tail(struct request_queue *q, struct request *rq)elv_rqhash_del (q, rq); q->nr_sorted--; +elv_fq_dispatched_request (q->elevator, rq); q->end_sector =rq_end_sector (rq); q->boundary_rq = rq;@@ -553,6 +564,7 @@ void elv_merge_requests(struct request_queue *q, struct request *rq,elv_rqhash_del (q, next); q->nr_sorted--; +elv_ioq_request_removed (e, next); q->last_merge = rq; }@@ -657,12 +669,8 @@ void elv_insert(struct request_queue *q, struct request *rq, int where) q->last_merge = rq; } -/* -* Some ioscheds (cfq) run q->request_fn directly, so -* rq cannot be accessed after calling -* elevator_add_req_fn. -*/ q->elevator->ops->elevator_add_req_fn (q, rq); +elv_ioq_request_add (q, rq);break ;case ELEVATOR_INSERT_REQUEUE :@@ -872,13 +880,12 @@ void elv_dequeue_request(struct request_queue *q, struct request *rq)int elv_queue_empty (struct request_queue *q) { -struct elevator_queue *e = q->elevator; -if (!list_empty (&q->queue_head))return 0 ; -if (e->ops->elevator_queue_empty_fn) -return e->ops->elevator_queue_empty_fn (q); +/* Hopefully nr_sorted works and no need to call queue_empty_fn */ +if (q->nr_sorted) +return 0 ;return 1 ; }@@ -953,8 +960,11 @@ void elv_completed_request(struct request_queue *q, struct request *rq)*/ if (blk_account_rq (rq)) { q->in_flight--; -if (blk_sorted_rq (rq) && e->ops->elevator_completed_req_fn) - e->ops->elevator_completed_req_fn (q, rq); +if (blk_sorted_rq (rq)) { +if (e->ops->elevator_completed_req_fn) + e->ops->elevator_completed_req_fn (q, rq); +elv_ioq_completed_request (q, rq); + } }/* @@ -1242,3 +1252,17 @@ struct request *elv_rb_latter_request(struct request_queue *q,return NULL ; }EXPORT_SYMBOL (elv_rb_latter_request); + +/* Get the io scheduler queue pointer. For cfq, it is stored in rq->ioq*/ +void *elv_get_sched_queue (struct request_queue *q,struct request *rq) +{ +return ioq_sched_queue (rq_ioq (rq)); +} +EXPORT_SYMBOL (elv_get_sched_queue); + +/* Select an ioscheduler queue to dispatch request from. */ +void *elv_select_sched_queue (struct request_queue *q,int force) +{ +return ioq_sched_queue (elv_fq_select_ioq (q, force)); +} +EXPORT_SYMBOL (elv_select_sched_queue);include/linux/blkdev.h +5
-0@@ -245,6 +245,11 @@ struct request {/* for bidi */ struct request *next_rq; + +#ifdef CONFIG_ELV_FAIR_QUEUING +/* io queue request belongs to */ +struct io_queue *ioq; +#endif };static inline unsigned short req_get_ioprio (struct request *req)include/linux/elevator.h +51
-0@@ -2,6 +2,7 @@#define _LINUX_ELEVATOR_H#include <linux/percpu.h> +#include "../../block/elevator-fq.h" #ifdef CONFIG_BLOCK @@ -29,6 +30,18 @@ typedef void (elevator_deactivate_req_fn) (struct request_queue *, struct requestypedef void *(elevator_init_fn ) (struct request_queue *);typedef void (elevator_exit_fn ) (struct elevator_queue *); +#ifdef CONFIG_ELV_FAIR_QUEUING +typedef void (elevator_free_sched_queue_fn ) (struct elevator_queue *,void *); +typedef void (elevator_active_ioq_set_fn ) (struct request_queue *,void *,int ); +typedef void (elevator_active_ioq_reset_fn ) (struct request_queue *,void *); +typedef void (elevator_arm_slice_timer_fn ) (struct request_queue *,void *); +typedef int (elevator_should_preempt_fn ) (struct request_queue *,void *, +struct request *); +typedef int (elevator_update_idle_window_fn ) (struct elevator_queue *,void *, +struct request *); +typedef struct io_queue * (elevator_close_cooperator_fn ) (struct request_queue *, +void *,int probe); +#endif struct elevator_ops {@@ -56,6 +69,17 @@ struct elevator_opselevator_init_fn *elevator_init_fn;elevator_exit_fn *elevator_exit_fn;void (*trim)(struct io_context *); + +#ifdef CONFIG_ELV_FAIR_QUEUING +elevator_free_sched_queue_fn *elevator_free_sched_queue_fn; +elevator_active_ioq_set_fn *elevator_active_ioq_set_fn; +elevator_active_ioq_reset_fn *elevator_active_ioq_reset_fn; + +elevator_arm_slice_timer_fn *elevator_arm_slice_timer_fn; +elevator_should_preempt_fn *elevator_should_preempt_fn; +elevator_update_idle_window_fn *elevator_update_idle_window_fn; +elevator_close_cooperator_fn *elevator_close_cooperator_fn; +#endif };#define ELV_NAME_MAX (16)@@ -76,6 +100,9 @@ struct elevator_typestruct elv_fs_entry *elevator_attrs;char elevator_name[ELV_NAME_MAX ];struct module *elevator_owner; +#ifdef CONFIG_ELV_FAIR_QUEUING +int elevator_features; +#endif };/* @@ -89,6 +116,10 @@ struct elevator_queuestruct elevator_type *elevator_type;struct mutex sysfs_lock;struct hlist_head *hash; +#ifdef CONFIG_ELV_FAIR_QUEUING +/* fair queuing data */ +struct elv_fq_data efqd; +#endif };/* @@ -209,5 +240,25 @@ enum { __val; \ }) +/* iosched can let elevator know their feature set/capability */ +#ifdef CONFIG_ELV_FAIR_QUEUING + +/* iosched wants to use fq logic of elevator layer */ +#define ELV_IOSCHED_NEED_FQ 1 + +static inline int elv_iosched_fair_queuing_enabled (struct elevator_queue *e) +{ +return (e->elevator_type->elevator_features) &ELV_IOSCHED_NEED_FQ ; +} + +#else /* ELV_IOSCHED_FAIR_QUEUING */ + +static inline int elv_iosched_fair_queuing_enabled (struct elevator_queue *e) +{ +return 0 ; +} +#endif /* ELV_IOSCHED_FAIR_QUEUING */ +extern void *elv_get_sched_queue (struct request_queue *q,struct request *rq); +extern void *elv_select_sched_queue (struct request_queue *q,int force);#endif /* CONFIG_BLOCK */ #endif Signature · 1 line
1.6.0.6Signature · 1 line
Balbir
Re: [PATCH 02/20] io-controller: Common flat fair queuing code in elevaotor layer
On Mon, Jun 22, 2009 at 02:43:13PM +0200, Fabio Checconi wrote:
[..]
Quoted 40 lines
Quoted 30 lines
Quoted 30 lines
block/elevator-fq.c new file +30
-0+/** +* bfq_first_active - find the eligible entity with the smallest finish time +* @st: the service tree to select from. +* +* This function searches the first schedulable entity, starting from the +* root of the tree and going on the left every time on this side there is +* a subtree with at least one eligible (start <= vtime) entity. The path +* on the right is followed only if a) the left subtree contains no eligible +* entities and b) no eligible entity has been found yet. +*/ +static struct io_entity *bfq_first_active_entity (struct io_service_tree *st) +{ +struct io_entity *entry, *first =NULL ; +struct rb_node *node = st->active.rb_node; + +while (node !=NULL ) { + entry = rb_entry(node, struct io_entity, rb_node); +left: +if (!bfq_gt (entry->start, st->vtime)) + first = entry; + +BUG_ON (bfq_gt (entry->min_start, st->vtime)); + +if (node->rb_left !=NULL ) { + entry =rb_entry (node->rb_left, + struct io_entity, rb_node); +if (!bfq_gt (entry->min_start, st->vtime)) { + node = node->rb_left; + goto left; + }Quoted 9 lines
block/elevator-fq.c line 701
Quoted 4 lines
block/elevator-fq.c new file +4
-0+ } +if (first !=NULL ) +break ; + node = node->rb_right;Please help me understand this, we sort the tree by finish time, but search by vtime,
start_time. The worst case could easily be O(N), right?
Quoted 10 lines
no, (again, the full answer is in the paper); the nice property of
min_startis that it partitions the tree in two regions, one with eligible entities and one without any of them. once we know that there is one eligible entity (checking themin_startat the root) we can find the node i withmin(F_i)subject toS_i< V walking down a single path from the root to the leftmost eligible entity. (we need to go to the right only if the subtree on the left contains no eligible entities at all.) since the RB tree is balanced this can be done in O(log N).
Hi Fabio,
When I go thorough the paper you mentioned above, they seem to have sorted the tree based on eligible time (looks like equivalent of start time) and then keep track of minimum deadline on each node (equivalnet of finish time).
We seem to be doing reverse in BFQ where we sort tree on finish time and keep track of minimum start time on each node. Is there any specific reason behind that?
Thanks
Vivek
Re: [PATCH 02/20] io-controller: Common flat fair queuing code in elevaotor layer
Quoted 5 lines
From: Vivek Goyal <[email protected]>
Date: Mon, Jun 22, 2009 10:43:37PM -0400On Mon, Jun 22, 2009 at 02:43:13PM +0200, Fabio Checconi wrote:
...
Quoted 18 lines
Quoted 5 lines
Quoted 4 lines
Please help me understand this, we sort the tree by finish time, but search by vtime,
start_time. The worst case could easily be O(N), right?Quoted 10 lines
no, (again, the full answer is in the paper); the nice property of
min_startis that it partitions the tree in two regions, one with eligible entities and one without any of them. once we know that there is one eligible entity (checking themin_startat the root) we can find the node i withmin(F_i)subject toS_i< V walking down a single path from the root to the leftmost eligible entity. (we need to go to the right only if the subtree on the left contains no eligible entities at all.) since the RB tree is balanced this can be done in O(log N).Hi Fabio,
Quoted 9 lines
When I go thorough the paper you mentioned above, they seem to have sorted the tree based on eligible time (looks like equivalent of start time) and then keep track of minimum deadline on each node (equivalnet of finish time).
We seem to be doing reverse in BFQ where we sort tree on finish time and keep track of minimum start time on each node. Is there any specific reason behind that?
Well... no specific reasons... I think that our implementation is easier to understand than the one of the paper, because it actually uses finish times as the ordering key, and min_start to quickly locate eligible subtrees, following the definition of the algorithm.
Moreover, if you look at the get_req() code in the paper, it needs a couple of loops to get to the result, while with our implementation we save the second loop.
Our version is still correct, because it always moves to the left (towards smaller finish times), except when moving to the left would mean entering a non feasible subtree, in which case it moves to the right.
Unfortunately I'm not aware of any paper describing a version of the algorithm more similar to the one we've implemented. Sorry for not having mentioned that difference in the comments nor anywhere else, it has been a long long time since I read the paper, and I must have forgotten about that.
Re: [PATCH 02/20] io-controller: Common flat fair queuing code in elevaotor layer
Quoted 34 lines · Fabio Checconi
Quoted 5 lines
From: Vivek Goyal <[email protected]>
Date: Mon, Jun 22, 2009 10:43:37PM -0400On Mon, Jun 22, 2009 at 02:43:13PM +0200, Fabio Checconi wrote:
...
Quoted 18 lines
Quoted 5 lines
Quoted 4 lines
Please help me understand this, we sort the tree by finish time, but search by vtime,
start_time. The worst case could easily be O(N), right?Quoted 10 lines
no, (again, the full answer is in the paper); the nice property of
min_startis that it partitions the tree in two regions, one with eligible entities and one without any of them. once we know that there is one eligible entity (checking themin_startat the root) we can find the node i withmin(F_i)subject toS_i< V walking down a single path from the root to the leftmost eligible entity. (we need to go to the right only if the subtree on the left contains no eligible entities at all.) since the RB tree is balanced this can be done in O(log N).Hi Fabio,
Quoted 9 lines
When I go thorough the paper you mentioned above, they seem to have sorted the tree based on eligible time (looks like equivalent of start time) and then keep track of minimum deadline on each node (equivalnet of finish time).
We seem to be doing reverse in BFQ where we sort tree on finish time and keep track of minimum start time on each node. Is there any specific reason behind that?
Quoted 5 lines
Well... no specific reasons... I think that our implementation is easier to understand than the one of the paper, because it actually uses finish times as the ordering key, and
min_startto quickly locate eligible subtrees, following the definition of the algorithm.
Is it still O(log N)?
Quoted 9 lines
Moreover, if you look at the
get_req()code in the paper, it needs a couple of loops to get to the result, while with our implementation we save the second loop.Our version is still correct, because it always moves to the left (towards smaller finish times), except when moving to the left would mean entering a non feasible subtree, in which case it moves to the right.
Quoted 5 lines
Unfortunately I'm not aware of any paper describing a version of the algorithm more similar to the one we've implemented. Sorry for not having mentioned that difference in the comments nor anywhere else, it has been a long long time since I read the paper, and I must have forgotten about that.
/me needs to go read the paper in full.
Signature · 1 line
Balbir
Re: [PATCH 02/20] io-controller: Common flat fair queuing code in elevaotor layer
Quoted 39 lines
From: Balbir Singh <[email protected]>
Date: Tue, Jun 23, 2009 01:02:52PM +0530Quoted 34 lines · Fabio Checconi
Quoted 5 lines
From: Vivek Goyal <[email protected]>
Date: Mon, Jun 22, 2009 10:43:37PM -0400On Mon, Jun 22, 2009 at 02:43:13PM +0200, Fabio Checconi wrote:
...
Quoted 27 lines
Quoted 5 lines
Quoted 4 lines
Please help me understand this, we sort the tree by finish time, but search by vtime,
start_time. The worst case could easily be O(N), right?Quoted 10 lines
no, (again, the full answer is in the paper); the nice property of
min_startis that it partitions the tree in two regions, one with eligible entities and one without any of them. once we know that there is one eligible entity (checking themin_startat the root) we can find the node i withmin(F_i)subject toS_i< V walking down a single path from the root to the leftmost eligible entity. (we need to go to the right only if the subtree on the left contains no eligible entities at all.) since the RB tree is balanced this can be done in O(log N).Hi Fabio,
When I go thorough the paper you mentioned above, they seem to have sorted the tree based on eligible time (looks like equivalent of start time) and then keep track of minimum deadline on each node (equivalnet of finish time).
We seem to be doing reverse in BFQ where we sort tree on finish time and keep track of minimum start time on each node. Is there any specific reason behind that?
Quoted 8 lines
Quoted 5 lines
Well... no specific reasons... I think that our implementation is easier to understand than the one of the paper, because it actually uses finish times as the ordering key, and
min_startto quickly locate eligible subtrees, following the definition of the algorithm.Is it still O(log N)?
Yes, it goes along a single path from the root to a leaf of a balanced tree (i.e., it starts from the root, and at the end of each iteration it selects the left or the right child of the current node), thus it is O(log N).
A message that is not in the archive (sent off-list or not yet received): 20090623073252.GJ8642-SINUvgVNF2CyUtPGxGje5AC/[email protected]
Re: [PATCH 02/20] io-controller: Common flat fair queuing code in elevaotor layer
Quoted 39 lines
From: Balbir Singh <[email protected]>
Date: Tue, Jun 23, 2009 01:02:52PM +0530Quoted 34 lines · Fabio Checconi
Quoted 5 lines
From: Vivek Goyal <[email protected]>
Date: Mon, Jun 22, 2009 10:43:37PM -0400On Mon, Jun 22, 2009 at 02:43:13PM +0200, Fabio Checconi wrote:
...
Quoted 27 lines
Quoted 5 lines
Quoted 4 lines
Please help me understand this, we sort the tree by finish time, but search by vtime,
start_time. The worst case could easily be O(N), right?Quoted 10 lines
no, (again, the full answer is in the paper); the nice property of
min_startis that it partitions the tree in two regions, one with eligible entities and one without any of them. once we know that there is one eligible entity (checking themin_startat the root) we can find the node i withmin(F_i)subject toS_i< V walking down a single path from the root to the leftmost eligible entity. (we need to go to the right only if the subtree on the left contains no eligible entities at all.) since the RB tree is balanced this can be done in O(log N).Hi Fabio,
When I go thorough the paper you mentioned above, they seem to have sorted the tree based on eligible time (looks like equivalent of start time) and then keep track of minimum deadline on each node (equivalnet of finish time).
We seem to be doing reverse in BFQ where we sort tree on finish time and keep track of minimum start time on each node. Is there any specific reason behind that?
Quoted 8 lines
Quoted 5 lines
Well... no specific reasons... I think that our implementation is easier to understand than the one of the paper, because it actually uses finish times as the ordering key, and
min_startto quickly locate eligible subtrees, following the definition of the algorithm.Is it still O(log N)?
Yes, it goes along a single path from the root to a leaf of a balanced tree (i.e., it starts from the root, and at the end of each iteration it selects the left or the right child of the current node), thus it is O(log N).
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: [PATCH 02/20] io-controller: Common flat fair queuing code in elevaotor layer
Quoted 34 lines · Fabio Checconi
Quoted 5 lines
From: Vivek Goyal <[email protected]>
Date: Mon, Jun 22, 2009 10:43:37PM -0400On Mon, Jun 22, 2009 at 02:43:13PM +0200, Fabio Checconi wrote:
...
Quoted 18 lines
Quoted 5 lines
Quoted 4 lines
Please help me understand this, we sort the tree by finish time, but search by vtime,
start_time. The worst case could easily be O(N), right?Quoted 10 lines
no, (again, the full answer is in the paper); the nice property of
min_startis that it partitions the tree in two regions, one with eligible entities and one without any of them. once we know that there is one eligible entity (checking themin_startat the root) we can find the node i withmin(F_i)subject toS_i< V walking down a single path from the root to the leftmost eligible entity. (we need to go to the right only if the subtree on the left contains no eligible entities at all.) since the RB tree is balanced this can be done in O(log N).Hi Fabio,
Quoted 9 lines
When I go thorough the paper you mentioned above, they seem to have sorted the tree based on eligible time (looks like equivalent of start time) and then keep track of minimum deadline on each node (equivalnet of finish time).
We seem to be doing reverse in BFQ where we sort tree on finish time and keep track of minimum start time on each node. Is there any specific reason behind that?
Quoted 5 lines
Well... no specific reasons... I think that our implementation is easier to understand than the one of the paper, because it actually uses finish times as the ordering key, and
min_startto quickly locate eligible subtrees, following the definition of the algorithm.
Is it still O(log N)?
Quoted 9 lines
Moreover, if you look at the
get_req()code in the paper, it needs a couple of loops to get to the result, while with our implementation we save the second loop.Our version is still correct, because it always moves to the left (towards smaller finish times), except when moving to the left would mean entering a non feasible subtree, in which case it moves to the right.
Quoted 5 lines
Unfortunately I'm not aware of any paper describing a version of the algorithm more similar to the one we've implemented. Sorry for not having mentioned that difference in the comments nor anywhere else, it has been a long long time since I read the paper, and I must have forgotten about that.
/me needs to go read the paper in full.
Signature · 1 line
Balbir
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: [PATCH 02/20] io-controller: Common flat fair queuing code in elevaotor layer
On Mon, Jun 22, 2009 at 02:43:13PM +0200, Fabio Checconi wrote:
[..]
Quoted 40 lines
Quoted 30 lines
Quoted 30 lines
block/elevator-fq.c new file +30
-0+/** +* bfq_first_active - find the eligible entity with the smallest finish time +* @st: the service tree to select from. +* +* This function searches the first schedulable entity, starting from the +* root of the tree and going on the left every time on this side there is +* a subtree with at least one eligible (start <= vtime) entity. The path +* on the right is followed only if a) the left subtree contains no eligible +* entities and b) no eligible entity has been found yet. +*/ +static struct io_entity *bfq_first_active_entity (struct io_service_tree *st) +{ +struct io_entity *entry, *first =NULL ; +struct rb_node *node = st->active.rb_node; + +while (node !=NULL ) { + entry = rb_entry(node, struct io_entity, rb_node); +left: +if (!bfq_gt (entry->start, st->vtime)) + first = entry; + +BUG_ON (bfq_gt (entry->min_start, st->vtime)); + +if (node->rb_left !=NULL ) { + entry =rb_entry (node->rb_left, + struct io_entity, rb_node); +if (!bfq_gt (entry->min_start, st->vtime)) { + node = node->rb_left; + goto left; + }Quoted 9 lines
block/elevator-fq.c line 701
Quoted 4 lines
block/elevator-fq.c new file +4
-0+ } +if (first !=NULL ) +break ; + node = node->rb_right;Please help me understand this, we sort the tree by finish time, but search by vtime,
start_time. The worst case could easily be O(N), right?
Quoted 10 lines
no, (again, the full answer is in the paper); the nice property of
min_startis that it partitions the tree in two regions, one with eligible entities and one without any of them. once we know that there is one eligible entity (checking themin_startat the root) we can find the node i withmin(F_i)subject toS_i< V walking down a single path from the root to the leftmost eligible entity. (we need to go to the right only if the subtree on the left contains no eligible entities at all.) since the RB tree is balanced this can be done in O(log N).
Hi Fabio,
When I go thorough the paper you mentioned above, they seem to have sorted the tree based on eligible time (looks like equivalent of start time) and then keep track of minimum deadline on each node (equivalnet of finish time).
We seem to be doing reverse in BFQ where we sort tree on finish time and keep track of minimum start time on each node. Is there any specific reason behind that?
Thanks
Vivek
A message that is not in the archive (sent off-list or not yet received): 20090622084612.GD3728-SINUvgVNF2CyUtPGxGje5AC/[email protected]
Re: [PATCH 02/20] io-controller: Common flat fair queuing code in elevaotor layer
Quoted 17 lines
From: Balbir Singh <[email protected]>
Date: Mon, Jun 22, 2009 02:16:12PM +0530Quoted 11 lines · Vivek Goyal
This is common fair queuing code in elevator layer. This is controlled by config option
CONFIG_ELV_FAIR_QUEUING. This patch initially only introduces flat fair queuing support where there is only one group, "root group" and all the tasks belong to root group.This elevator layer changes are backward compatible. That means any ioscheduler using old interfaces will continue to work.
This code is essentially the CFQ code for fair queuing. The primary difference is that flat rounding robin algorithm of CFQ has been replaced with BFQ (WF2Q+).
Quoted 6 lines
The patch is quite long and to be honest requires a long time to review, which I don't mind. I suspect my frequently diverted mind is likely to miss a lot in a big patch like this. Could you consider splitting this further if possible. I think you'll notice the number of reviews will also increase.
This core scheduler part has not changed too much from the bfq patches, so I'll try to answer your questions; Vivek, please correct me where my knowledge is outdated. I preferred to leave out the questions about code that was not in the original patches.
...
Quoted 37 lines
block/elevator-fq.c line 35
Quoted 2 lines
block/elevator-fq.c new file +2
-0+static inline int elv_prio_slice (struct elv_fq_data *efqd,int sync, +unsigned short prio)Why is the return type int and not unsigned int or unsigned long? Can the return value ever be negative?
Quoted 31 lines
block/elevator-fq.c new file +31
-0+{ +const int base_slice = efqd->elv_slice[sync]; + +WARN_ON (prio >=IOPRIO_BE_NR ); + +return base_slice + (base_slice/ELV_SLICE_SCALE * (4 - prio)); +} + +static inline int +elv_prio_to_slice (struct elv_fq_data *efqd,struct io_queue *ioq) +{ +return elv_prio_slice (efqd,elv_ioq_sync (ioq), ioq->entity.ioprio); +} + +/* Mainly the BFQ scheduling code Follows */ + +/* +* Shift for timestamp calculations. This actually limits the maximum +* service allowed in one timestamp delta (small shift values increase it), +* the maximum total weight that can be used for the queues in the system +* (big shift values increase it), and the period of virtual time wraparounds. +*/ +#define WFQ_SERVICE_SHIFT 22 + +/** +* bfq_gt - compare two timestamps. +* @a: first ts. +* @b: second ts. +* +* Return @a > @b, dealing with wrapping correctly. +*/
Quoted 9 lines
block/elevator-fq.c line 71
Quoted 5 lines
block/elevator-fq.c new file +5
-0+static inline int bfq_gt (bfq_timestamp_t a,bfq_timestamp_t b) +{ +return (s64)(a - b) >0 ; +} +a and b are of type u64, but cast to s64 to deal with wrapping? Correct?
yes
Quoted 5 lines
Quoted 5 lines
+/**
+ *bfq_delta- map service into the virtual time domain.
+ * @service: amount of service.
+ * @weight: scale factor.
+ */
Quoted 9 lines
block/elevator-fq.c line 81
Quoted 5 lines
block/elevator-fq.c new file +5
-0+static inline bfq_timestamp_t bfq_delta (bfq_service_t service, +bfq_weight_t weight) +{ +bfq_timestamp_t d = (bfq_timestamp_t )service <<WFQ_SERVICE_SHIFT ; +Why is the case required? Does the compiler complain? service is already of the correct type.
service is unsigned long, so it can be 32 bits in 32 bit machines, while timestamps are always u64, so I think we need the cast.
Quoted 6 lines
block/elevator-fq.c line 82
Quoted 1 line
block/elevator-fq.c new file +1
-0+do_div (d, weight);On a 64 system both d and weight are 64 bit, but on a 32 bit system weight is 32 bits.
do_divexpects a 64 bit dividend and 32 bit divisor
- no?
yes. here the situation is that we actually don't care about the type of weight, as long as it can contain a 32 bit value, and weights should never reach near the 2^32 boundary, otherwise we're prone to any kind of numerical error. there are no problems with weight being u32.
Quoted 10 lines
Quoted 10 lines
block/elevator-fq.c new file +10
-0+return d; +} + +/** +* bfq_calc_finish - assign the finish time to an entity. +* @entity: the entity to act upon. +* @service: the service to be charged to the entity. +*/ +static inline void bfq_calc_finish (struct io_entity *entity, +bfq_service_t service)
Quoted 9 lines
block/elevator-fq.c line 97
Quoted 5 lines
block/elevator-fq.c new file +5
-0+{ +BUG_ON (entity->weight ==0 ); + + entity->finish = entity->start +bfq_delta (service, entity->weight); +}Should we
BUG_ON(entity->finish==entity->start)? Or is that expected when the entity has no service time left.
bfq_calc_finish() is used in two cases:
1) we need to resync the finish time with the service received by an entity
2) we need to assign a new finish time to an entity when it's enqueued
with preemptions 1) can happen with service = 0, and we need to reset the finish time to the start time (depending on how preemptions are implemented), so in this case we'd have a false positive (leading to a crashed system :) ).
Quoted 40 lines
Quoted 27 lines
block/elevator-fq.c new file +27
-0+ +static inline struct io_queue *io_entity_to_ioq (struct io_entity *entity) +{ +struct io_queue *ioq =NULL ; + +BUG_ON (entity ==NULL ); +if (entity->my_sched_data ==NULL ) + ioq = container_of(entity, struct io_queue, entity); +return ioq; +} + +/** +* bfq_entity_of - get an entity from a node. +* @node: the node field of the entity. +* +* Convert a node pointer to the relative entity. This is used only +* to simplify the logic of some functions and not as the generic +* conversion mechanism because, e.g., in the tree walking functions, +* the check for a %NULL value would be redundant. +*/ +static inline struct io_entity *bfq_entity_of (struct rb_node *node) +{ +struct io_entity *entity =NULL ; + +if (node !=NULL ) + entity = rb_entry(node, struct io_entity, rb_node); +block/elevator-fq.c line 134
Quoted 10 lines
block/elevator-fq.c new file +10
-0+return entity; +} + +/** +* bfq_extract - remove an entity from a tree. +* @root: the tree root. +* @entity: the entity to remove. +*/ +static inline void bfq_extract (struct rb_root *root,struct io_entity *entity) +{Extract is not common terminology, why not use
bfq_remove()?
Quoted 7 lines
block/elevator-fq.c line 138
Quoted 4 lines
block/elevator-fq.c new file +4
-0+BUG_ON (entity->tree != root); + + entity->tree =NULL ; +rb_erase (&entity->rb_node, root);Don't you want to make
entity->tree= NULL afterrb_erase?
this code assumes to be executed under spinlock, so order doesn't really matter (tree is not affected by rb_erase(), it is a bfq private field).
Quoted 10 lines
Quoted 10 lines
block/elevator-fq.c new file +10
-0+} + +/** +* bfq_idle_extract - extract an entity from the idle tree. +* @st: the service tree of the owning @entity. +* @entity: the entity being removed. +*/ +static void bfq_idle_extract (struct io_service_tree *st, +struct io_entity *entity) +{
Quoted 9 lines
block/elevator-fq.c line 154
Quoted 6 lines
block/elevator-fq.c new file +6
-0+struct rb_node *next; + +BUG_ON (entity->tree != &st->idle); + +if (entity == st->first_idle) { + next =rb_next (&entity->rb_node);What happens if next is NULL?
the bfq_entity_of() call below returns NULL
Quoted 8 lines
block/elevator-fq.c line 159
Quoted 5 lines
block/elevator-fq.c new file +5
-0+ st->first_idle =bfq_entity_of (next); + } + +if (entity == st->last_idle) { + next =rb_prev (&entity->rb_node);What happens if next is NULL?
same as above
Quoted 55 lines
Quoted 55 lines
block/elevator-fq.c new file +55
-0+ st->last_idle =bfq_entity_of (next); + } + +bfq_extract (&st->idle, entity); +} + +/** +* bfq_insert - generic tree insertion. +* @root: tree root. +* @entity: entity to insert. +* +* This is used for the idle and the active tree, since they are both +* ordered by finish time. +*/ +static void bfq_insert (struct rb_root *root,struct io_entity *entity) +{ +struct io_entity *entry; +struct rb_node **node = &root->rb_node; +struct rb_node *parent =NULL ; + +BUG_ON (entity->tree !=NULL ); + +while (*node !=NULL ) { + parent = *node; + entry = rb_entry(parent, struct io_entity, rb_node); + +if (bfq_gt (entry->finish, entity->finish)) + node = &parent->rb_left; +else + node = &parent->rb_right; + } + +rb_link_node (&entity->rb_node, parent, node); +rb_insert_color (&entity->rb_node, root); + + entity->tree = root; +} + +/** +* bfq_update_min - update the min_start field of a entity. +* @entity: the entity to update. +* @node: one of its children. +* +* This function is called when @entity may store an invalid value for +* min_start due to updates to the active tree. The function assumes +* that the subtree rooted at @node (which may be its left or its right +* child) has a valid min_start value. +*/ +static inline void bfq_update_min (struct io_entity *entity, +struct rb_node *node) +{ +struct io_entity *child; + +if (node !=NULL ) { + child = rb_entry(node, struct io_entity, rb_node);
Quoted 9 lines
block/elevator-fq.c line 218
Quoted 4 lines
block/elevator-fq.c new file +4
-0+if (bfq_gt (entity->min_start, child->min_start)) + entity->min_start = child->min_start; + } +}So.. we check to see if child's
min_timeis lesser than the root entities or node entities and set it to the minimum of the two? Can you usemin_there?
no, it would not deal with wraparound correctly
Quoted 13 lines
Quoted 13 lines
block/elevator-fq.c new file +13
-0+ +/** +* bfq_update_active_node - recalculate min_start. +* @node: the node to update. +* +* @node may have changed position or one of its children may have moved, +* this function updates its min_start value. The left and right subtrees +* are assumed to hold a correct min_start value. +*/ +static inline void bfq_update_active_node (struct rb_node *node) +{ + struct io_entity *entity = rb_entry(node, struct io_entity, rb_node); +
Quoted 9 lines
block/elevator-fq.c line 235
Quoted 4 lines
block/elevator-fq.c new file +4
-0+ entity->min_start = entity->start; +bfq_update_min (entity, node->rb_right); +bfq_update_min (entity, node->rb_left); +}I don't like this every much, we set the
min_timetwice, this can be easily optimized to look at both left and right child and pick the minimum.
it's a minimum between three values (the ->start fields of the two children and of the node itself), you cannot be sure it will be set twice
Quoted 26 lines
Quoted 26 lines
block/elevator-fq.c new file +26
-0+ +/** +* bfq_update_active_tree - update min_start for the whole active tree. +* @node: the starting node. +* +* @node must be the deepest modified node after an update. This function +* updates its min_start using the values held by its children, assuming +* that they did not change, and then updates all the nodes that may have +* changed in the path to the root. The only nodes that may have changed +* are the ones in the path or their siblings. +*/ +static void bfq_update_active_tree (struct rb_node *node) +{ +struct rb_node *parent; + +up: +bfq_update_active_node (node); + + parent =rb_parent (node); +if (parent ==NULL ) +return ; + +if (node == parent->rb_left && parent->rb_right !=NULL ) +bfq_update_active_node (parent->rb_right); +else if (parent->rb_left !=NULL ) +bfq_update_active_node (parent->rb_left);
Quoted 9 lines
block/elevator-fq.c line 266
Quoted 5 lines
block/elevator-fq.c new file +5
-0+ + node = parent; + goto up; +} +For these functions, take a look at the walk function in the group scheduler that does
update_shares
are you sure? AFAICT walk_tg_tree() walks all over the tree, this just walks a single path to a node up to the root, I don't see what the two have in common.
in the original patches we cited (among the others):
http://www.cs.berkeley.edu/~istoica/papers/eevdf-tr-95.pdf
which contains a description of the algorithm.
Quoted 59 lines
Quoted 59 lines
block/elevator-fq.c new file +59
-0+/** +* bfq_active_insert - insert an entity in the active tree of its group/device. +* @st: the service tree of the entity. +* @entity: the entity being inserted. +* +* The active tree is ordered by finish time, but an extra key is kept +* per each node, containing the minimum value for the start times of +* its children (and the node itself), so it's possible to search for +* the eligible node with the lowest finish time in logarithmic time. +*/ +static void bfq_active_insert (struct io_service_tree *st, +struct io_entity *entity) +{ +struct rb_node *node = &entity->rb_node; + +bfq_insert (&st->active, entity); + +if (node->rb_left !=NULL ) + node = node->rb_left; +else if (node->rb_right !=NULL ) + node = node->rb_right; + +bfq_update_active_tree (node); +} + +/** +* bfq_ioprio_to_weight - calc a weight from an ioprio. +* @ioprio: the ioprio value to convert. +*/ +static bfq_weight_t bfq_ioprio_to_weight (int ioprio) +{ +WARN_ON (ioprio <0 || ioprio >=IOPRIO_BE_NR ); +return IOPRIO_BE_NR - ioprio; +} + +void bfq_get_entity (struct io_entity *entity) +{ +struct io_queue *ioq =io_entity_to_ioq (entity); + +if (ioq) +elv_get_ioq (ioq); +} + +void bfq_init_entity (struct io_entity *entity,struct io_group *iog) +{ + entity->ioprio = entity->new_ioprio; + entity->ioprio_class = entity->new_ioprio_class; + entity->sched_data = &iog->sched_data; +} + +/** +* bfq_find_deepest - find the deepest node that an extraction can modify. +* @node: the node being removed. +* +* Do the first step of an extraction in an rb tree, looking for the +* node that will replace @node, and returning the deepest node that +* the following modifications to the tree can touch. If @node is the +* last node in the tree return %NULL. +*/
Quoted 9 lines
block/elevator-fq.c line 331
Quoted 6 lines
block/elevator-fq.c new file +6
-0+static struct rb_node *bfq_find_deepest (struct rb_node *node) +{ +struct rb_node *deepest; + +if (node->rb_right ==NULL && node->rb_left ==NULL ) + deepest =rb_parent (node);Why is the parent the deepest? Shouldn't node be the deepest?
this is related to how the RB tree is updated (see below)
Quoted 9 lines
Quoted 9 lines
block/elevator-fq.c new file +9
-0+else if (node->rb_right ==NULL ) + deepest = node->rb_left; +else if (node->rb_left ==NULL ) + deepest = node->rb_right; +else { + deepest =rb_next (node); +if (deepest->rb_right !=NULL ) + deepest = deepest->rb_right; +else if (rb_parent (deepest) != node)
Quoted 9 lines
block/elevator-fq.c line 345
Quoted 5 lines
block/elevator-fq.c new file +5
-0+ deepest =rb_parent (deepest); + } + +return deepest; +}The function is not clear, could you please define deepest node better?
according to the paper cited above, we need to update the min_start value on the path from the deepest node modified by the extraction up to the root. this function tries to consider all the cases of RB extraction, looking for the deepest node that (after all the rotations etc.) will need an update to min_start. one interesting property of RB trees is that this can be done in O(log N) because there is a single path that needs to be updated.
Quoted 13 lines
Quoted 13 lines
block/elevator-fq.c new file +13
-0+ +/** +* bfq_active_extract - remove an entity from the active tree. +* @st: the service_tree containing the tree. +* @entity: the entity being removed. +*/ +static void bfq_active_extract (struct io_service_tree *st, +struct io_entity *entity) +{ +struct rb_node *node; + + node =bfq_find_deepest (&entity->rb_node); +bfq_extract (&st->active, entity);
Quoted 9 lines
block/elevator-fq.c line 363
Quoted 5 lines
block/elevator-fq.c new file +5
-0+ +if (node !=NULL ) +bfq_update_active_tree (node); +} +Just to check my understanding, every time an active node is added/removed, we update the
min_timeof the entire tree.
yes, but only O(log N) nodes need to be updated
Quoted 334 lines
Quoted 334 lines
block/elevator-fq.c new file +334
-0+/** +* bfq_idle_insert - insert an entity into the idle tree. +* @st: the service tree containing the tree. +* @entity: the entity to insert. +*/ +static void bfq_idle_insert (struct io_service_tree *st, +struct io_entity *entity) +{ +struct io_entity *first_idle = st->first_idle; +struct io_entity *last_idle = st->last_idle; + +if (first_idle ==NULL ||bfq_gt (first_idle->finish, entity->finish)) + st->first_idle = entity; +if (last_idle ==NULL ||bfq_gt (entity->finish, last_idle->finish)) + st->last_idle = entity; + +bfq_insert (&st->idle, entity); +} + +/** +* bfq_forget_entity - remove an entity from the wfq trees. +* @st: the service tree. +* @entity: the entity being removed. +* +* Update the device status and forget everything about @entity, putting +* the device reference to it, if it is a queue. Entities belonging to +* groups are not refcounted. +*/ +static void bfq_forget_entity (struct io_service_tree *st, +struct io_entity *entity) +{ +struct io_queue *ioq =NULL ; + +BUG_ON (!entity->on_st); + entity->on_st =0 ; + st->wsum -= entity->weight; + ioq =io_entity_to_ioq (entity); +if (!ioq) +return ; +elv_put_ioq (ioq); +} + +/** +* bfq_put_idle_entity - release the idle tree ref of an entity. +* @st: service tree for the entity. +* @entity: the entity being released. +*/ +void bfq_put_idle_entity (struct io_service_tree *st, +struct io_entity *entity) +{ +bfq_idle_extract (st, entity); +bfq_forget_entity (st, entity); +} + +/** +* bfq_forget_idle - update the idle tree if necessary. +* @st: the service tree to act upon. +* +* To preserve the global O(log N) complexity we only remove one entry here; +* as the idle tree will not grow indefinitely this can be done safely. +*/ +void bfq_forget_idle (struct io_service_tree *st) +{ +struct io_entity *first_idle = st->first_idle; +struct io_entity *last_idle = st->last_idle; + +if (RB_EMPTY_ROOT (&st->active) && last_idle !=NULL && + !bfq_gt (last_idle->finish, st->vtime)) { +/* +* Active tree is empty. Pull back vtime to finish time of +* last idle entity on idle tree. +* Rational seems to be that it reduces the possibility of +* vtime wraparound (bfq_gt(V-F) < 0). +*/ + st->vtime = last_idle->finish; + } + +if (first_idle !=NULL && !bfq_gt (first_idle->finish, st->vtime)) +bfq_put_idle_entity (st, first_idle); +} + + +static struct io_service_tree * +__bfq_entity_update_prio (struct io_service_tree *old_st, +struct io_entity *entity) +{ +struct io_service_tree *new_st = old_st; +struct io_queue *ioq =io_entity_to_ioq (entity); + +if (entity->ioprio_changed) { + entity->ioprio = entity->new_ioprio; + entity->ioprio_class = entity->new_ioprio_class; + entity->ioprio_changed =0 ; + +/* +* Also update the scaled budget for ioq. Group will get the +* updated budget once ioq is selected to run next. +*/ +if (ioq) { +struct elv_fq_data *efqd = ioq->efqd; + entity->budget =elv_prio_to_slice (efqd, ioq); + } + + old_st->wsum -= entity->weight; + entity->weight =bfq_ioprio_to_weight (entity->ioprio); + +/* +* NOTE: here we may be changing the weight too early, +* this will cause unfairness. The correct approach +* would have required additional complexity to defer +* weight changes to the proper time instants (i.e., +* when entity->finish <= old_st->vtime). +*/ + new_st =io_entity_service_tree (entity); + new_st->wsum += entity->weight; + +if (new_st != old_st) + entity->start = new_st->vtime; + } + +return new_st; +} + +/** +* __bfq_activate_entity - activate an entity. +* @entity: the entity being activated. +* +* Called whenever an entity is activated, i.e., it is not active and one +* of its children receives a new request, or has to be reactivated due to +* budget exhaustion. It uses the current budget of the entity (and the +* service received if @entity is active) of the queue to calculate its +* timestamps. +*/ +static void __bfq_activate_entity (struct io_entity *entity,int add_front) +{ +struct io_sched_data *sd = entity->sched_data; +struct io_service_tree *st =io_entity_service_tree (entity); + +if (entity == sd->active_entity) { +BUG_ON (entity->tree !=NULL ); +/* +* If we are requeueing the current entity we have +* to take care of not charging to it service it has +* not received. +*/ +bfq_calc_finish (entity, entity->service); + entity->start = entity->finish; + sd->active_entity =NULL ; + }else if (entity->tree == &st->active) { +/* +* Requeueing an entity due to a change of some +* next_active entity below it. We reuse the old +* start time. +*/ +bfq_active_extract (st, entity); + }else if (entity->tree == &st->idle) { +/* +* Must be on the idle tree, bfq_idle_extract() will +* check for that. +*/ +bfq_idle_extract (st, entity); + entity->start =bfq_gt (st->vtime, entity->finish) ? + st->vtime : entity->finish; + }else { +/* +* The finish time of the entity may be invalid, and +* it is in the past for sure, otherwise the queue +* would have been on the idle tree. +*/ + entity->start = st->vtime; + st->wsum += entity->weight; +bfq_get_entity (entity); + +BUG_ON (entity->on_st); + entity->on_st =1 ; + } + + st =__bfq_entity_update_prio (st, entity); +/* +* This is to emulate cfq like functionality where preemption can +* happen with-in same class, like sync queue preempting async queue +* May be this is not a very good idea from fairness point of view +* as preempting queue gains share. Keeping it for now. +*/ +if (add_front) { +struct io_entity *next_entity; + +/* +* Determine the entity which will be dispatched next +* Use sd->next_active once hierarchical patch is applied +*/ + next_entity =bfq_lookup_next_entity (sd,0 ); + +if (next_entity && next_entity != entity) { +struct io_service_tree *new_st; +bfq_timestamp_t delta; + + new_st =io_entity_service_tree (next_entity); + +/* +* At this point, both entities should belong to +* same service tree as cross service tree preemption +* is automatically taken care by algorithm +*/ +BUG_ON (new_st != st); + entity->finish = next_entity->finish -1 ; + delta =bfq_delta (entity->budget, entity->weight); + entity->start = entity->finish - delta; +if (bfq_gt (entity->start, st->vtime)) + entity->start = st->vtime; + } + }else { +bfq_calc_finish (entity, entity->budget); + } +bfq_active_insert (st, entity); +} + +/** +* bfq_activate_entity - activate an entity. +* @entity: the entity to activate. +*/ +void bfq_activate_entity (struct io_entity *entity,int add_front) +{ +__bfq_activate_entity (entity, add_front); +} + +/** +* __bfq_deactivate_entity - deactivate an entity from its service tree. +* @entity: the entity to deactivate. +* @requeue: if false, the entity will not be put into the idle tree. +* +* Deactivate an entity, independently from its previous state. If the +* entity was not on a service tree just return, otherwise if it is on +* any scheduler tree, extract it from that tree, and if necessary +* and if the caller did not specify @requeue, put it on the idle tree. +* +*/ +int __bfq_deactivate_entity (struct io_entity *entity,int requeue) +{ +struct io_sched_data *sd = entity->sched_data; +struct io_service_tree *st =io_entity_service_tree (entity); +int was_active = entity == sd->active_entity; +int ret =0 ; + +if (!entity->on_st) +return 0 ; + +BUG_ON (was_active && entity->tree !=NULL ); + +if (was_active) { +bfq_calc_finish (entity, entity->service); + sd->active_entity =NULL ; + }else if (entity->tree == &st->active) +bfq_active_extract (st, entity); +else if (entity->tree == &st->idle) +bfq_idle_extract (st, entity); +else if (entity->tree !=NULL ) +BUG (); + +if (!requeue || !bfq_gt (entity->finish, st->vtime)) +bfq_forget_entity (st, entity); +else +bfq_idle_insert (st, entity); + +BUG_ON (sd->active_entity == entity); + +return ret; +} + +/** +* bfq_deactivate_entity - deactivate an entity. +* @entity: the entity to deactivate. +* @requeue: true if the entity can be put on the idle tree +*/ +void bfq_deactivate_entity (struct io_entity *entity,int requeue) +{ +__bfq_deactivate_entity (entity, requeue); +} + +/** +* bfq_update_vtime - update vtime if necessary. +* @st: the service tree to act upon. +* +* If necessary update the service tree vtime to have at least one +* eligible entity, skipping to its start time. Assumes that the +* active tree of the device is not empty. +* +* NOTE: this hierarchical implementation updates vtimes quite often, +* we may end up with reactivated tasks getting timestamps after a +* vtime skip done because we needed a ->first_active entity on some +* intermediate node. +*/ +static void bfq_update_vtime (struct io_service_tree *st) +{ +struct io_entity *entry; +struct rb_node *node = st->active.rb_node; + + entry = rb_entry(node, struct io_entity, rb_node); +if (bfq_gt (entry->min_start, st->vtime)) { + st->vtime = entry->min_start; +bfq_forget_idle (st); + } +} + +/** +* bfq_first_active - find the eligible entity with the smallest finish time +* @st: the service tree to select from. +* +* This function searches the first schedulable entity, starting from the +* root of the tree and going on the left every time on this side there is +* a subtree with at least one eligible (start <= vtime) entity. The path +* on the right is followed only if a) the left subtree contains no eligible +* entities and b) no eligible entity has been found yet. +*/ +static struct io_entity *bfq_first_active_entity (struct io_service_tree *st) +{ +struct io_entity *entry, *first =NULL ; +struct rb_node *node = st->active.rb_node; + +while (node !=NULL ) { + entry = rb_entry(node, struct io_entity, rb_node); +left: +if (!bfq_gt (entry->start, st->vtime)) + first = entry; + +BUG_ON (bfq_gt (entry->min_start, st->vtime)); + +if (node->rb_left !=NULL ) { + entry =rb_entry (node->rb_left, + struct io_entity, rb_node); +if (!bfq_gt (entry->min_start, st->vtime)) { + node = node->rb_left; + goto left; + }
Quoted 9 lines
block/elevator-fq.c line 701
Quoted 4 lines
block/elevator-fq.c new file +4
-0+ } +if (first !=NULL ) +break ; + node = node->rb_right;Please help me understand this, we sort the tree by finish time, but search by vtime,
start_time. The worst case could easily be O(N), right?
no, (again, the full answer is in the paper); the nice property of min_start is that it partitions the tree in two regions, one with eligible entities and one without any of them. once we know that there is one eligible entity (checking the min_start at the root) we can find the node i with min(F_i) subject to S_i < V walking down a single path from the root to the leftmost eligible entity. (we need to go to the right only if the subtree on the left contains no eligible entities at all.) since the RB tree is balanced this can be done in O(log N).
Quoted 69 lines
Quoted 69 lines
block/elevator-fq.c new file +69
-0+ } + +BUG_ON (first ==NULL && !RB_EMPTY_ROOT (&st->active)); +return first; +} + +/** +* __bfq_lookup_next_entity - return the first eligible entity in @st. +* @st: the service tree. +* +* Update the virtual time in @st and return the first eligible entity +* it contains. +*/ +static struct io_entity *__bfq_lookup_next_entity (struct io_service_tree *st) +{ +struct io_entity *entity; + +if (RB_EMPTY_ROOT (&st->active)) +return NULL ; + +bfq_update_vtime (st); + entity =bfq_first_active_entity (st); +BUG_ON (bfq_gt (entity->start, st->vtime)); + +return entity; +} + +/** +* bfq_lookup_next_entity - return the first eligible entity in @sd. +* @sd: the sched_data. +* @extract: if true the returned entity will be also extracted from @sd. +* +* NOTE: since we cache the next_active entity at each level of the +* hierarchy, the complexity of the lookup can be decreased with +* absolutely no effort just returning the cached next_active value; +* we prefer to do full lookups to test the consistency of * the data +* structures. +*/ +struct io_entity *bfq_lookup_next_entity (struct io_sched_data *sd, +int extract) +{ +struct io_service_tree *st = sd->service_tree; +struct io_entity *entity; +int i; + +/* +* We should not call lookup when an entity is active, as doing lookup +* can result in an erroneous vtime jump. +*/ +BUG_ON (sd->active_entity !=NULL ); + +for (i =0 ; i <IO_IOPRIO_CLASSES ; i++, st++) { + entity =__bfq_lookup_next_entity (st); +if (entity !=NULL ) { +if (extract) { +bfq_active_extract (st, entity); + sd->active_entity = entity; + } +break ; + } + } + +return entity; +} + +void entity_served (struct io_entity *entity,bfq_service_t served) +{ +struct io_service_tree *st; +
Quoted 9 lines
block/elevator-fq.c line 775
Quoted 5 lines
block/elevator-fq.c new file +5
-0+ st =io_entity_service_tree (entity); + entity->service += served; +BUG_ON (st->wsum ==0 ); + st->vtime +=bfq_delta (served, st->wsum); +bfq_forget_idle (st);Forget idle checks to see if the
st->vtime>first_idle->finish, if so it pushes thefirst_idleto later, right?
yes, updating the weight sum accordingly
Quoted 182 lines
Quoted 182 lines
block/elevator-fq.c new file +182
-0+} + +/** +* bfq_flush_idle_tree - deactivate any entity on the idle tree of @st. +* @st: the service tree being flushed. +*/ +void io_flush_idle_tree (struct io_service_tree *st) +{ +struct io_entity *entity = st->first_idle; + +for (; entity !=NULL ; entity = st->first_idle) +__bfq_deactivate_entity (entity,0 ); +} + +/* Elevator fair queuing function */ +struct io_queue *rq_ioq (struct request *rq) +{ +return rq->ioq; +} + +static inline struct io_queue *elv_active_ioq (struct elevator_queue *e) +{ +return e->efqd.active_queue; +} + +void *elv_active_sched_queue (struct elevator_queue *e) +{ +return ioq_sched_queue (elv_active_ioq (e)); +} +EXPORT_SYMBOL (elv_active_sched_queue); + +int elv_nr_busy_ioq (struct elevator_queue *e) +{ +return e->efqd.busy_queues; +} +EXPORT_SYMBOL (elv_nr_busy_ioq); + +int elv_hw_tag (struct elevator_queue *e) +{ +return e->efqd.hw_tag; +} +EXPORT_SYMBOL (elv_hw_tag); + +/* Helper functions for operating on elevator idle slice timer */ +int elv_mod_idle_slice_timer (struct elevator_queue *eq,unsigned long expires) +{ +struct elv_fq_data *efqd = &eq->efqd; + +return mod_timer (&efqd->idle_slice_timer, expires); +} +EXPORT_SYMBOL (elv_mod_idle_slice_timer); + +int elv_del_idle_slice_timer (struct elevator_queue *eq) +{ +struct elv_fq_data *efqd = &eq->efqd; + +return del_timer (&efqd->idle_slice_timer); +} +EXPORT_SYMBOL (elv_del_idle_slice_timer); + +unsigned int elv_get_slice_idle (struct elevator_queue *eq) +{ +return eq->efqd.elv_slice_idle; +} +EXPORT_SYMBOL (elv_get_slice_idle); + +void elv_ioq_served (struct io_queue *ioq,bfq_service_t served) +{ +entity_served (&ioq->entity, served); +} + +/* Tells whether ioq is queued in root group or not */ +static inline int is_root_group_ioq (struct request_queue *q, +struct io_queue *ioq) +{ +struct elv_fq_data *efqd = &q->elevator->efqd; + +return (ioq->entity.sched_data == &efqd->root_group->sched_data); +} + +/* +* sysfs parts below --> +*/ +static ssize_t +elv_var_show (unsigned int var,char *page) +{ +return sprintf (page,"%d\n" , var); +} + +static ssize_t +elv_var_store (unsigned int *var,const char *page,size_t count) +{ +char *p = (char *) page; + + *var =simple_strtoul (p, &p,10 ); +return count; +} + +#define SHOW_FUNCTION (__FUNC, __VAR, __CONV) \ +ssize_t __FUNC(struct elevator_queue *e, char *page) \ +{ \ + struct elv_fq_data *efqd = &e->efqd; \ + unsigned int __data = __VAR; \ + if (__CONV) \ + __data = jiffies_to_msecs(__data); \ + return elv_var_show(__data, (page)); \ +} +SHOW_FUNCTION (elv_slice_idle_show, efqd->elv_slice_idle,1 ); +EXPORT_SYMBOL (elv_slice_idle_show); +SHOW_FUNCTION (elv_slice_sync_show, efqd->elv_slice[1 ],1 ); +EXPORT_SYMBOL (elv_slice_sync_show); +SHOW_FUNCTION (elv_slice_async_show, efqd->elv_slice[0 ],1 ); +EXPORT_SYMBOL (elv_slice_async_show); +#undef SHOW_FUNCTION + +#define STORE_FUNCTION (__FUNC, __PTR,MIN ,MAX , __CONV) \ +ssize_t __FUNC(struct elevator_queue *e, const char *page, size_t count)\ +{ \ + struct elv_fq_data *efqd = &e->efqd; \ + unsigned int __data; \ + int ret = elv_var_store(&__data, (page), count); \ + if (__data < (MIN)) \ + __data = (MIN); \ + else if (__data > (MAX)) \ + __data = (MAX); \ + if (__CONV) \ + *(__PTR) = msecs_to_jiffies(__data); \ + else \ + *(__PTR) = __data; \ + return ret; \ +} +STORE_FUNCTION (elv_slice_idle_store, &efqd->elv_slice_idle,0 ,UINT_MAX ,1 ); +EXPORT_SYMBOL (elv_slice_idle_store); +STORE_FUNCTION (elv_slice_sync_store, &efqd->elv_slice[1 ],1 ,UINT_MAX ,1 ); +EXPORT_SYMBOL (elv_slice_sync_store); +STORE_FUNCTION (elv_slice_async_store, &efqd->elv_slice[0 ],1 ,UINT_MAX ,1 ); +EXPORT_SYMBOL (elv_slice_async_store); +#undef STORE_FUNCTION + +void elv_schedule_dispatch (struct request_queue *q) +{ +struct elv_fq_data *efqd = &q->elevator->efqd; + +if (elv_nr_busy_ioq (q->elevator)) { +elv_log (efqd,"schedule dispatch" ); +kblockd_schedule_work (efqd->queue, &efqd->unplug_work); + } +} +EXPORT_SYMBOL (elv_schedule_dispatch); + +void elv_kick_queue (struct work_struct *work) +{ +struct elv_fq_data *efqd = + container_of(work, struct elv_fq_data, unplug_work); +struct request_queue *q = efqd->queue; +unsigned long flags; + +spin_lock_irqsave (q->queue_lock, flags); +blk_start_queueing (q); +spin_unlock_irqrestore (q->queue_lock, flags); +} + +void elv_shutdown_timer_wq (struct elevator_queue *e) +{ +del_timer_sync (&e->efqd.idle_slice_timer); +cancel_work_sync (&e->efqd.unplug_work); +} +EXPORT_SYMBOL (elv_shutdown_timer_wq); + +void elv_ioq_set_prio_slice (struct request_queue *q,struct io_queue *ioq) +{ +struct elv_fq_data *efqd = &q->elevator->efqd; + + ioq->slice_end = jiffies + ioq->entity.budget; +elv_log_ioq (efqd, ioq,"set_slice=%lu" , ioq->entity.budget); +} + +static void elv_ioq_update_io_thinktime (struct io_queue *ioq) +{ +struct elv_fq_data *efqd = ioq->efqd; +unsigned long elapsed = jiffies - ioq->last_end_request; +unsigned long ttime =min (elapsed,2UL * efqd->elv_slice_idle);
Quoted 9 lines
block/elevator-fq.c line 962
Quoted 5 lines
block/elevator-fq.c new file +5
-0+ + ioq->ttime_samples = (7 *ioq->ttime_samples +256 ) /8 ; + ioq->ttime_total = (7 *ioq->ttime_total +256 *ttime) /8 ; + ioq->ttime_mean = (ioq->ttime_total +128 ) / ioq->ttime_samples; +}Not sure I understand the magical 7, 8, 2, 128 and 256. Please help me understand the algorithm.
this came from cfq, it's a variation of an exponential moving average, with ttime_samples used to scale the average value.
Quoted 1,083 lines
Quoted 68 lines
block/elevator-fq.c new file +68
-0+ +/* +* Disable idle window if the process thinks too long. +* This idle flag can also be updated by io scheduler. +*/ +static void elv_ioq_update_idle_window (struct elevator_queue *eq, +struct io_queue *ioq,struct request *rq) +{ +int old_idle, enable_idle; +struct elv_fq_data *efqd = ioq->efqd; + +/* +* Don't idle for async or idle io prio class +*/ +if (!elv_ioq_sync (ioq) ||elv_ioq_class_idle (ioq)) +return ; + + enable_idle = old_idle =elv_ioq_idle_window (ioq); + +if (!efqd->elv_slice_idle) + enable_idle =0 ; +else if (ioq_sample_valid (ioq->ttime_samples)) { +if (ioq->ttime_mean > efqd->elv_slice_idle) + enable_idle =0 ; +else + enable_idle =1 ; + } + +/* +* From think time perspective idle should be enabled. Check with +* io scheduler if it wants to disable idling based on additional +* considrations like seek pattern. +*/ +if (enable_idle) { +if (eq->ops->elevator_update_idle_window_fn) + enable_idle = eq->ops->elevator_update_idle_window_fn ( + eq, ioq->sched_queue, rq); +if (!enable_idle) +elv_log_ioq (efqd, ioq,"iosched disabled idle" ); + } + +if (old_idle != enable_idle) { +elv_log_ioq (efqd, ioq,"idle=%d" , enable_idle); +if (enable_idle) +elv_mark_ioq_idle_window (ioq); +else +elv_clear_ioq_idle_window (ioq); + } +} + +struct io_queue *elv_alloc_ioq (struct request_queue *q,gfp_t gfp_mask) +{ +struct io_queue *ioq =NULL ; + + ioq =kmem_cache_alloc_node (elv_ioq_pool, gfp_mask, q->node); +return ioq; +} +EXPORT_SYMBOL (elv_alloc_ioq); + +void elv_free_ioq (struct io_queue *ioq) +{ +kmem_cache_free (elv_ioq_pool, ioq); +} +EXPORT_SYMBOL (elv_free_ioq); + +int elv_init_ioq (struct elevator_queue *eq,struct io_queue *ioq, +void *sched_queue,int ioprio_class,int ioprio, +int is_sync)block/elevator-fq.c line 1040
Quoted 10 lines
block/elevator-fq.c new file +10
-0+{ +struct elv_fq_data *efqd = &eq->efqd; +struct io_group *iog =io_lookup_io_group_current (efqd->queue); + +RB_CLEAR_NODE (&ioq->entity.rb_node); +atomic_set (&ioq->ref,0 ); + ioq->efqd = efqd; +elv_ioq_set_ioprio_class (ioq, ioprio_class); +elv_ioq_set_ioprio (ioq, ioprio); + ioq->pid = current->pid;Is pid used for cgroup association later? I don't see why we save the pid otherwise? If yes, why not store the cgroup of the
current->pid?Quoted 95 lines
block/elevator-fq.c new file +95
-0+ ioq->sched_queue = sched_queue; +if (is_sync && !elv_ioq_class_idle (ioq)) +elv_mark_ioq_idle_window (ioq); +bfq_init_entity (&ioq->entity, iog); + ioq->entity.budget =elv_prio_to_slice (efqd, ioq); +if (is_sync) + ioq->last_end_request = jiffies; + +return 0 ; +} +EXPORT_SYMBOL (elv_init_ioq); + +void elv_put_ioq (struct io_queue *ioq) +{ +struct elv_fq_data *efqd = ioq->efqd; + struct elevator_queue *e = container_of(efqd, struct elevator_queue, + efqd); + +BUG_ON (atomic_read (&ioq->ref) <=0 ); +if (!atomic_dec_and_test (&ioq->ref)) +return ; +BUG_ON (ioq->nr_queued); +BUG_ON (ioq->entity.tree !=NULL ); +BUG_ON (elv_ioq_busy (ioq)); +BUG_ON (efqd->active_queue == ioq); + +/* Can be called by outgoing elevator. Don't use q */ +BUG_ON (!e->ops->elevator_free_sched_queue_fn); + + e->ops->elevator_free_sched_queue_fn (e, ioq->sched_queue); +elv_log_ioq (efqd, ioq,"put_queue" ); +elv_free_ioq (ioq); +} +EXPORT_SYMBOL (elv_put_ioq); + +void elv_release_ioq (struct elevator_queue *e,struct io_queue **ioq_ptr) +{ +struct io_queue *ioq = *ioq_ptr; + +if (ioq !=NULL ) { +/* Drop the reference taken by the io group */ +elv_put_ioq (ioq); + *ioq_ptr =NULL ; + } +} + +/* +* Normally next io queue to be served is selected from the service tree. +* This function allows one to choose a specific io queue to run next +* out of order. This is primarily to accomodate the close_cooperator +* feature of cfq. +* +* Currently it is done only for root level as to begin with supporting +* close cooperator feature only for root group to make sure default +* cfq behavior in flat hierarchy is not changed. +*/ +void elv_set_next_ioq (struct request_queue *q,struct io_queue *ioq) +{ +struct elv_fq_data *efqd = &q->elevator->efqd; +struct io_entity *entity = &ioq->entity; +struct io_sched_data *sd = &efqd->root_group->sched_data; +struct io_service_tree *st =io_entity_service_tree (entity); + +BUG_ON (efqd->active_queue !=NULL || sd->active_entity !=NULL ); +BUG_ON (!efqd->busy_queues); +BUG_ON (sd != entity->sched_data); +BUG_ON (!st); + +bfq_update_vtime (st); +bfq_active_extract (st, entity); + sd->active_entity = entity; + entity->service =0 ; +elv_log_ioq (efqd, ioq,"set_next_ioq" ); +} + +/* Get next queue for service. */ +struct io_queue *elv_get_next_ioq (struct request_queue *q,int extract) +{ +struct elv_fq_data *efqd = &q->elevator->efqd; +struct io_entity *entity =NULL ; +struct io_queue *ioq =NULL ; +struct io_sched_data *sd; + +/* +* We should not call lookup when an entity is active, as doing +* lookup can result in an erroneous vtime jump. +*/ +BUG_ON (efqd->active_queue !=NULL ); + +if (!efqd->busy_queues) +return NULL ; + + sd = &efqd->root_group->sched_data; + entity =bfq_lookup_next_entity (sd,1 ); +block/elevator-fq.c line 1145
Quoted 10 lines
block/elevator-fq.c new file +10
-0+BUG_ON (!entity); +if (extract) + entity->service =0 ; + ioq =io_entity_to_ioq (entity); + +return ioq; +} + +/* +* coop tells that io scheduler selected a queue for us and we did not coop?
Quoted 893 lines
+ * select the next queue based on fairness.
+ */block/elevator-fq.c new file +868
-0+static void __elv_set_active_ioq (struct elv_fq_data *efqd,struct io_queue *ioq, +int coop) +{ +struct request_queue *q = efqd->queue; + +if (ioq) { +elv_log_ioq (efqd, ioq,"set_active, busy=%d" , + efqd->busy_queues); + ioq->slice_end =0 ; + +elv_clear_ioq_wait_request (ioq); +elv_clear_ioq_must_dispatch (ioq); +elv_mark_ioq_slice_new (ioq); + +del_timer (&efqd->idle_slice_timer); + } + + efqd->active_queue = ioq; + +/* Let iosched know if it wants to take some action */ +if (ioq) { +if (q->elevator->ops->elevator_active_ioq_set_fn) + q->elevator->ops->elevator_active_ioq_set_fn (q, + ioq->sched_queue, coop); + } +} + +/* Get and set a new active queue for service. */ +struct io_queue *elv_set_active_ioq (struct request_queue *q, +struct io_queue *ioq) +{ +struct elv_fq_data *efqd = &q->elevator->efqd; +int coop =0 ; + +if (!ioq) + ioq =elv_get_next_ioq (q,1 ); +else { +elv_set_next_ioq (q, ioq); +/* +* io scheduler selected the next queue for us. Pass this +* this info back to io scheudler. cfq currently uses it +* to reset coop flag on the queue. +*/ + coop =1 ; + } +__elv_set_active_ioq (efqd, ioq, coop); +return ioq; +} + +void elv_reset_active_ioq (struct elv_fq_data *efqd) +{ +struct request_queue *q = efqd->queue; +struct io_queue *ioq =elv_active_ioq (efqd->queue->elevator); + +if (q->elevator->ops->elevator_active_ioq_reset_fn) + q->elevator->ops->elevator_active_ioq_reset_fn (q, + ioq->sched_queue); + efqd->active_queue =NULL ; +del_timer (&efqd->idle_slice_timer); +} + +void elv_activate_ioq (struct io_queue *ioq,int add_front) +{ +bfq_activate_entity (&ioq->entity, add_front); +} + +void elv_deactivate_ioq (struct elv_fq_data *efqd,struct io_queue *ioq, +int requeue) +{ +bfq_deactivate_entity (&ioq->entity, requeue); +} + +/* Called when an inactive queue receives a new request. */ +void elv_add_ioq_busy (struct elv_fq_data *efqd,struct io_queue *ioq) +{ +BUG_ON (elv_ioq_busy (ioq)); +BUG_ON (ioq == efqd->active_queue); +elv_log_ioq (efqd, ioq,"add to busy" ); +elv_activate_ioq (ioq,0 ); +elv_mark_ioq_busy (ioq); + efqd->busy_queues++; +if (elv_ioq_class_rt (ioq)) { +struct io_group *iog =ioq_to_io_group (ioq); + iog->busy_rt_queues++; + } +} + +void elv_del_ioq_busy (struct elevator_queue *e,struct io_queue *ioq, +int requeue) +{ +struct elv_fq_data *efqd = &e->efqd; + +BUG_ON (!elv_ioq_busy (ioq)); +BUG_ON (ioq->nr_queued); +elv_log_ioq (efqd, ioq,"del from busy" ); +elv_clear_ioq_busy (ioq); +BUG_ON (efqd->busy_queues ==0 ); + efqd->busy_queues--; +if (elv_ioq_class_rt (ioq)) { +struct io_group *iog =ioq_to_io_group (ioq); + iog->busy_rt_queues--; + } + +elv_deactivate_ioq (efqd, ioq, requeue); +} + +/* +* Do the accounting. Determine how much service (in terms of time slices) +* current queue used and adjust the start, finish time of queue and vtime +* of the tree accordingly. +* +* Determining the service used in terms of time is tricky in certain +* situations. Especially when underlying device supports command queuing +* and requests from multiple queues can be there at same time, then it +* is not clear which queue consumed how much of disk time. +* +* To mitigate this problem, cfq starts the time slice of the queue only +* after first request from the queue has completed. This does not work +* very well if we expire the queue before we wait for first and more +* request to finish from the queue. For seeky queues, we will expire the +* queue after dispatching few requests without waiting and start dispatching +* from next queue. +* +* Not sure how to determine the time consumed by queue in such scenarios. +* Currently as a crude approximation, we are charging 25% of time slice +* for such cases. A better mechanism is needed for accurate accounting. +*/ +void __elv_ioq_slice_expired (struct request_queue *q,struct io_queue *ioq) +{ +struct elv_fq_data *efqd = &q->elevator->efqd; +struct io_entity *entity = &ioq->entity; +long slice_unused =0 , slice_used =0 , slice_overshoot =0 ; + +assert_spin_locked (q->queue_lock); +elv_log_ioq (efqd, ioq,"slice expired" ); + +if (elv_ioq_wait_request (ioq)) +del_timer (&efqd->idle_slice_timer); + +elv_clear_ioq_wait_request (ioq); + +/* +* if ioq->slice_end = 0, that means a queue was expired before first +* reuqest from the queue got completed. Of course we are not planning +* to idle on the queue otherwise we would not have expired it. +* +* Charge for the 25% slice in such cases. This is not the best thing +* to do but at the same time not very sure what's the next best +* thing to do. +* +* This arises from that fact that we don't have the notion of +* one queue being operational at one time. io scheduler can dispatch +* requests from multiple queues in one dispatch round. Ideally for +* more accurate accounting of exact disk time used by disk, one +* should dispatch requests from only one queue and wait for all +* the requests to finish. But this will reduce throughput. +*/ +if (!ioq->slice_end) + slice_used = entity->budget/4 ; +else { +if (time_after (ioq->slice_end, jiffies)) { + slice_unused = ioq->slice_end - jiffies; +if (slice_unused == entity->budget) { +/* +* queue got expired immediately after +* completing first request. Charge 25% of +* slice. +*/ + slice_used = entity->budget/4 ; + }else + slice_used = entity->budget - slice_unused; + }else { + slice_overshoot = jiffies - ioq->slice_end; + slice_used = entity->budget + slice_overshoot; + } + } + +elv_log_ioq (efqd, ioq,"sl_end=%lx, jiffies=%lx" , ioq->slice_end, + jiffies); +elv_log_ioq (efqd, ioq,"sl_used=%ld, budget=%ld overshoot=%ld" , + slice_used, entity->budget, slice_overshoot); +elv_ioq_served (ioq, slice_used); + +BUG_ON (ioq != efqd->active_queue); +elv_reset_active_ioq (efqd); + +if (!ioq->nr_queued) +elv_del_ioq_busy (q->elevator, ioq,1 ); +else +elv_activate_ioq (ioq,0 ); +} +EXPORT_SYMBOL (__elv_ioq_slice_expired); + +/* +* Expire the ioq. +*/ +void elv_ioq_slice_expired (struct request_queue *q) +{ +struct io_queue *ioq =elv_active_ioq (q->elevator); + +if (ioq) +__elv_ioq_slice_expired (q, ioq); +} + +/* +* Check if new_cfqq should preempt the currently active queue. Return 0 for +* no or if we aren't sure, a 1 will cause a preemption attempt. +*/ +int elv_should_preempt (struct request_queue *q,struct io_queue *new_ioq, +struct request *rq) +{ +struct io_queue *ioq; +struct elevator_queue *eq = q->elevator; +struct io_entity *entity, *new_entity; + + ioq =elv_active_ioq (eq); + +if (!ioq) +return 0 ; + + entity = &ioq->entity; + new_entity = &new_ioq->entity; + +/* +* Allow an RT request to pre-empt an ongoing non-RT cfqq timeslice. +*/ + +if (new_entity->ioprio_class ==IOPRIO_CLASS_RT + && entity->ioprio_class !=IOPRIO_CLASS_RT ) +return 1 ; +/* +* Allow an BE request to pre-empt an ongoing IDLE clas timeslice. +*/ + +if (new_entity->ioprio_class ==IOPRIO_CLASS_BE + && entity->ioprio_class ==IOPRIO_CLASS_IDLE ) +return 1 ; + +/* +* Check with io scheduler if it has additional criterion based on +* which it wants to preempt existing queue. +*/ +if (eq->ops->elevator_should_preempt_fn) +return eq->ops->elevator_should_preempt_fn (q, +ioq_sched_queue (new_ioq), rq); + +return 0 ; +} + +static void elv_preempt_queue (struct request_queue *q,struct io_queue *ioq) +{ +elv_log_ioq (&q->elevator->efqd, ioq,"preempt" ); +elv_ioq_slice_expired (q); + +/* +* Put the new queue at the front of the of the current list, +* so we know that it will be selected next. +*/ + +elv_activate_ioq (ioq,1 ); +elv_ioq_set_slice_end (ioq,0 ); +elv_mark_ioq_slice_new (ioq); +} + +void elv_ioq_request_add (struct request_queue *q,struct request *rq) +{ +struct elv_fq_data *efqd = &q->elevator->efqd; +struct io_queue *ioq = rq->ioq; + +if (!elv_iosched_fair_queuing_enabled (q->elevator)) +return ; + +BUG_ON (!efqd); +BUG_ON (!ioq); + efqd->rq_queued++; + ioq->nr_queued++; + +if (!elv_ioq_busy (ioq)) +elv_add_ioq_busy (efqd, ioq); + +elv_ioq_update_io_thinktime (ioq); +elv_ioq_update_idle_window (q->elevator, ioq, rq); + +if (ioq ==elv_active_ioq (q->elevator)) { +/* +* Remember that we saw a request from this process, but +* don't start queuing just yet. Otherwise we risk seeing lots +* of tiny requests, because we disrupt the normal plugging +* and merging. If the request is already larger than a single +* page, let it rip immediately. For that case we assume that +* merging is already done. Ditto for a busy system that +* has other work pending, don't risk delaying until the +* idle timer unplug to continue working. +*/ +if (elv_ioq_wait_request (ioq)) { +if (blk_rq_bytes (rq) >PAGE_CACHE_SIZE || + efqd->busy_queues >1 ) { +del_timer (&efqd->idle_slice_timer); +blk_start_queueing (q); + } +elv_mark_ioq_must_dispatch (ioq); + } + }else if (elv_should_preempt (q, ioq, rq)) { +/* +* not the active queue - expire current slice if it is +* idle and has expired it's mean thinktime or this new queue +* has some old slice time left and is of higher priority or +* this new queue is RT and the current one is BE +*/ +elv_preempt_queue (q, ioq); +blk_start_queueing (q); + } +} + +void elv_idle_slice_timer (unsigned long data) +{ +struct elv_fq_data *efqd = (struct elv_fq_data *)data; +struct io_queue *ioq; +unsigned long flags; +struct request_queue *q = efqd->queue; + +elv_log (efqd,"idle timer fired" ); + +spin_lock_irqsave (q->queue_lock, flags); + + ioq = efqd->active_queue; + +if (ioq) { + +/* +* We saw a request before the queue expired, let it through +*/ +if (elv_ioq_must_dispatch (ioq)) + goto out_kick; + +/* +* expired +*/ +if (elv_ioq_slice_used (ioq)) + goto expire; + +/* +* only expire and reinvoke request handler, if there are +* other queues with pending requests +*/ +if (!elv_nr_busy_ioq (q->elevator)) + goto out_cont; + +/* +* not expired and it has a request pending, let it dispatch +*/ +if (ioq->nr_queued) + goto out_kick; + } +expire: +elv_ioq_slice_expired (q); +out_kick: +elv_schedule_dispatch (q); +out_cont: +spin_unlock_irqrestore (q->queue_lock, flags); +} + +void elv_ioq_arm_slice_timer (struct request_queue *q) +{ +struct elv_fq_data *efqd = &q->elevator->efqd; +struct io_queue *ioq =elv_active_ioq (q->elevator); +unsigned long sl; + +BUG_ON (!ioq); + +/* +* SSD device without seek penalty, disable idling. But only do so +* for devices that support queuing, otherwise we still have a problem +* with sync vs async workloads. +*/ +if (blk_queue_nonrot (q) && efqd->hw_tag) +return ; + +/* +* still requests with the driver, don't idle +*/ +if (efqd->rq_in_driver) +return ; + +/* +* idle is disabled, either manually or by past process history +*/ +if (!efqd->elv_slice_idle || !elv_ioq_idle_window (ioq)) +return ; + +/* +* may be iosched got its own idling logic. In that case io +* schduler will take care of arming the timer, if need be. +*/ +if (q->elevator->ops->elevator_arm_slice_timer_fn) { + q->elevator->ops->elevator_arm_slice_timer_fn (q, + ioq->sched_queue); + }else { +elv_mark_ioq_wait_request (ioq); + sl = efqd->elv_slice_idle; +mod_timer (&efqd->idle_slice_timer, jiffies + sl); +elv_log_ioq (efqd, ioq,"arm idle: %lu" , sl); + } +} + +/* Common layer function to select the next queue to dispatch from */ +void *elv_fq_select_ioq (struct request_queue *q,int force) +{ +struct elv_fq_data *efqd = &q->elevator->efqd; +struct io_queue *new_ioq =NULL , *ioq =elv_active_ioq (q->elevator); +struct io_group *iog; + +if (!elv_nr_busy_ioq (q->elevator)) +return NULL ; + +if (ioq ==NULL ) + goto new_queue; + +/* +* Force dispatch. Continue to dispatch from current queue as long +* as it has requests. +*/ +if (unlikely (force)) { +if (ioq->nr_queued) + goto keep_queue; +else + goto expire; + } + +/* +* The active queue has run out of time, expire it and select new. +*/ +if (elv_ioq_slice_used (ioq) && !elv_ioq_must_dispatch (ioq)) + goto expire; + +/* +* If we have a RT cfqq waiting, then we pre-empt the current non-rt +* cfqq. +*/ + iog =ioq_to_io_group (ioq); + +if (!elv_ioq_class_rt (ioq) && iog->busy_rt_queues) { +/* +* We simulate this as cfqq timed out so that it gets to bank +* the remaining of its time slice. +*/ +elv_log_ioq (efqd, ioq,"preempt" ); + goto expire; + } + +/* +* The active queue has requests and isn't expired, allow it to +* dispatch. +*/ + +if (ioq->nr_queued) + goto keep_queue; + +/* +* If another queue has a request waiting within our mean seek +* distance, let it run. The expire code will check for close +* cooperators and put the close queue at the front of the service +* tree. +*/ + new_ioq =elv_close_cooperator (q, ioq,0 ); +if (new_ioq) + goto expire; + +/* +* No requests pending. If the active queue still has requests in +* flight or is idling for a new request, allow either of these +* conditions to happen (or time out) before selecting a new queue. +*/ + +if (timer_pending (&efqd->idle_slice_timer) || + (elv_ioq_nr_dispatched (ioq) &&elv_ioq_idle_window (ioq))) { + ioq =NULL ; + goto keep_queue; + } + +expire: +elv_ioq_slice_expired (q); +new_queue: + ioq =elv_set_active_ioq (q, new_ioq); +keep_queue: +return ioq; +} + +/* A request got removed from io_queue. Do the accounting */ +void elv_ioq_request_removed (struct elevator_queue *e,struct request *rq) +{ +struct io_queue *ioq; +struct elv_fq_data *efqd; + +if (!elv_iosched_fair_queuing_enabled (e)) +return ; + + ioq = rq->ioq; +BUG_ON (!ioq); + ioq->nr_queued--; + + efqd = ioq->efqd; +BUG_ON (!efqd); + efqd->rq_queued--; + +if (elv_ioq_busy (ioq) && (elv_active_ioq (e) != ioq) && !ioq->nr_queued) +elv_del_ioq_busy (e, ioq,1 ); +} + +/* A request got dispatched. Do the accounting. */ +void elv_fq_dispatched_request (struct elevator_queue *e,struct request *rq) +{ +struct io_queue *ioq = rq->ioq; + +if (!elv_iosched_fair_queuing_enabled (e)) +return ; + +BUG_ON (!ioq); +elv_ioq_request_dispatched (ioq); +elv_ioq_request_removed (e, rq); +elv_clear_ioq_must_dispatch (ioq); +} + +void elv_fq_activate_rq (struct request_queue *q,struct request *rq) +{ +struct elv_fq_data *efqd = &q->elevator->efqd; + +if (!elv_iosched_fair_queuing_enabled (q->elevator)) +return ; + + efqd->rq_in_driver++; +elv_log_ioq (efqd,rq_ioq (rq),"activate rq, drv=%d" , + efqd->rq_in_driver); +} + +void elv_fq_deactivate_rq (struct request_queue *q,struct request *rq) +{ +struct elv_fq_data *efqd = &q->elevator->efqd; + +if (!elv_iosched_fair_queuing_enabled (q->elevator)) +return ; + +WARN_ON (!efqd->rq_in_driver); + efqd->rq_in_driver--; +elv_log_ioq (efqd,rq_ioq (rq),"deactivate rq, drv=%d" , + efqd->rq_in_driver); +} + +/* +* Update hw_tag based on peak queue depth over 50 samples under +* sufficient load. +*/ +static void elv_update_hw_tag (struct elv_fq_data *efqd) +{ +if (efqd->rq_in_driver > efqd->rq_in_driver_peak) + efqd->rq_in_driver_peak = efqd->rq_in_driver; + +if (efqd->rq_queued <=ELV_HW_QUEUE_MIN && + efqd->rq_in_driver <=ELV_HW_QUEUE_MIN ) +return ; + +if (efqd->hw_tag_samples++ <50 ) +return ; + +if (efqd->rq_in_driver_peak >=ELV_HW_QUEUE_MIN ) + efqd->hw_tag =1 ; +else + efqd->hw_tag =0 ; + + efqd->hw_tag_samples =0 ; + efqd->rq_in_driver_peak =0 ; +} + +/* +* If ioscheduler has functionality of keeping track of close cooperator, check +* with it if it has got a closely co-operating queue. +*/ +static inline struct io_queue *elv_close_cooperator (struct request_queue *q, +struct io_queue *ioq,int probe) +{ +struct elevator_queue *e = q->elevator; +struct io_queue *new_ioq =NULL ; + +/* +* Currently this feature is supported only for flat hierarchy or +* root group queues so that default cfq behavior is not changed. +*/ +if (!is_root_group_ioq (q, ioq)) +return NULL ; + +if (q->elevator->ops->elevator_close_cooperator_fn) + new_ioq = e->ops->elevator_close_cooperator_fn (q, + ioq->sched_queue, probe); + +/* Only select co-operating queue if it belongs to root group */ +if (new_ioq && !is_root_group_ioq (q, new_ioq)) +return NULL ; + +return new_ioq; +} + +/* A request got completed from io_queue. Do the accounting. */ +void elv_ioq_completed_request (struct request_queue *q,struct request *rq) +{ +const int sync =rq_is_sync (rq); +struct io_queue *ioq; +struct elv_fq_data *efqd = &q->elevator->efqd; + +if (!elv_iosched_fair_queuing_enabled (q->elevator)) +return ; + + ioq = rq->ioq; + +elv_log_ioq (efqd, ioq,"complete" ); + +elv_update_hw_tag (efqd); + +WARN_ON (!efqd->rq_in_driver); +WARN_ON (!ioq->dispatched); + efqd->rq_in_driver--; + ioq->dispatched--; + +if (sync) + ioq->last_end_request = jiffies; + +/* +* If this is the active queue, check if it needs to be expired, +* or if we want to idle in case it has no pending requests. +*/ + +if (elv_active_ioq (q->elevator) == ioq) { +if (elv_ioq_slice_new (ioq)) { +elv_ioq_set_prio_slice (q, ioq); +elv_clear_ioq_slice_new (ioq); + } +/* +* If there are no requests waiting in this queue, and +* there are other queues ready to issue requests, AND +* those other queues are issuing requests within our +* mean seek distance, give them a chance to run instead +* of idling. +*/ +if (elv_ioq_slice_used (ioq) ||elv_ioq_class_idle (ioq)) +elv_ioq_slice_expired (q); +else if (!ioq->nr_queued && !elv_close_cooperator (q, ioq,1 ) + && sync && !rq_noidle (rq)) +elv_ioq_arm_slice_timer (q); + } + +if (!efqd->rq_in_driver) +elv_schedule_dispatch (q); +} + +struct io_group *io_lookup_io_group_current (struct request_queue *q) +{ +struct elv_fq_data *efqd = &q->elevator->efqd; + +return efqd->root_group; +} +EXPORT_SYMBOL (io_lookup_io_group_current); + +void *io_group_async_queue_prio (struct io_group *iog,int ioprio_class, +int ioprio) +{ +struct io_queue *ioq =NULL ; + +switch (ioprio_class) { +case IOPRIO_CLASS_RT : + ioq = iog->async_queue[0 ][ioprio]; +break ; +case IOPRIO_CLASS_BE : + ioq = iog->async_queue[1 ][ioprio]; +break ; +case IOPRIO_CLASS_IDLE : + ioq = iog->async_idle_queue; +break ; +default : +BUG (); + } + +if (ioq) +return ioq->sched_queue; +return NULL ; +} +EXPORT_SYMBOL (io_group_async_queue_prio); + +void io_group_set_async_queue (struct io_group *iog,int ioprio_class, +int ioprio,struct io_queue *ioq) +{ +switch (ioprio_class) { +case IOPRIO_CLASS_RT : + iog->async_queue[0 ][ioprio] = ioq; +break ; +case IOPRIO_CLASS_BE : + iog->async_queue[1 ][ioprio] = ioq; +break ; +case IOPRIO_CLASS_IDLE : + iog->async_idle_queue = ioq; +break ; +default : +BUG (); + } + +/* +* Take the group reference and pin the queue. Group exit will +* clean it up +*/ +elv_get_ioq (ioq); +} +EXPORT_SYMBOL (io_group_set_async_queue); + +/* +* Release all the io group references to its async queues. +*/ +void io_put_io_group_queues (struct elevator_queue *e,struct io_group *iog) +{ +int i, j; + +for (i =0 ; i <2 ; i++) +for (j =0 ; j <IOPRIO_BE_NR ; j++) +elv_release_ioq (e, &iog->async_queue[i][j]); + +/* Free up async idle queue */ +elv_release_ioq (e, &iog->async_idle_queue); +} + +struct io_group *io_alloc_root_group (struct request_queue *q, +struct elevator_queue *e,void *key) +{ +struct io_group *iog; +int i; + + iog =kmalloc_node (sizeof (*iog),GFP_KERNEL | __GFP_ZERO, q->node); +if (iog ==NULL ) +return NULL ; + +for (i =0 ; i <IO_IOPRIO_CLASSES ; i++) + iog->sched_data.service_tree[i] =IO_SERVICE_TREE_INIT ; + +return iog; +} + +void io_free_root_group (struct elevator_queue *e) +{ +struct io_group *iog = e->efqd.root_group; +struct io_service_tree *st; +int i; + +for (i =0 ; i <IO_IOPRIO_CLASSES ; i++) { + st = iog->sched_data.service_tree + i; +io_flush_idle_tree (st); + } + +io_put_io_group_queues (e, iog); +kfree (iog); +} + +static void elv_slab_kill (void ) +{ +/* +* Caller already ensured that pending RCU callbacks are completed, +* so we should have no busy allocations at this point. +*/ +if (elv_ioq_pool) +kmem_cache_destroy (elv_ioq_pool); +} + +static int __init elv_slab_setup (void ) +{ + elv_ioq_pool =KMEM_CACHE (io_queue,0 ); +if (!elv_ioq_pool) + goto fail; + +return 0 ; +fail: +elv_slab_kill (); +return -ENOMEM ; +} + +/* Initialize fair queueing data associated with elevator */ +int elv_init_fq_data (struct request_queue *q,struct elevator_queue *e) +{ +struct io_group *iog; +struct elv_fq_data *efqd = &e->efqd; + +if (!elv_iosched_fair_queuing_enabled (e)) +return 0 ; + + iog =io_alloc_root_group (q, e, efqd); +if (iog ==NULL ) +return 1 ; + + efqd->root_group = iog; + efqd->queue = q; + +init_timer (&efqd->idle_slice_timer); + efqd->idle_slice_timer.function = elv_idle_slice_timer; + efqd->idle_slice_timer.data = (unsigned long ) efqd; + +INIT_WORK (&efqd->unplug_work, elv_kick_queue); + + efqd->elv_slice[0 ] = elv_slice_async; + efqd->elv_slice[1 ] = elv_slice_sync; + efqd->elv_slice_idle = elv_slice_idle; + efqd->hw_tag =1 ; + +return 0 ; +} + +/* +* elv_exit_fq_data is called before we call elevator_exit_fn. Before +* we ask elevator to cleanup its queues, we do the cleanup here so +* that all the group and idle tree references to ioq are dropped. Later +* during elevator cleanup, ioc reference will be dropped which will lead +* to removal of ioscheduler queue as well as associated ioq object. +*/ +void elv_exit_fq_data (struct elevator_queue *e) +{ +struct elv_fq_data *efqd = &e->efqd; + +if (!elv_iosched_fair_queuing_enabled (e)) +return ; + +elv_shutdown_timer_wq (e); + +BUG_ON (timer_pending (&efqd->idle_slice_timer)); +io_free_root_group (e); +} + +/* +* This is called after the io scheduler has cleaned up its data structres. +* I don't think that this function is required. Right now just keeping it +* because cfq cleans up timer and work queue again after freeing up +* io contexts. To me io scheduler has already been drained out, and all +* the active queue have already been expired so time and work queue should +* not been activated during cleanup process. +* +* Keeping it here for the time being. Will get rid of it later. +*/ +void elv_exit_fq_data_post (struct elevator_queue *e) +{ +struct elv_fq_data *efqd = &e->efqd; + +if (!elv_iosched_fair_queuing_enabled (e)) +return ; + +elv_shutdown_timer_wq (e); +BUG_ON (timer_pending (&efqd->idle_slice_timer)); +} + + +static int __init elv_fq_init (void ) +{ +if (elv_slab_setup ()) +return -ENOMEM ; + +/* could be 0 on HZ < 1000 setups */ + +if (!elv_slice_async) + elv_slice_async =1 ; + +if (!elv_slice_idle) + elv_slice_idle =1 ; + +return 0 ; +} + +module_init (elv_fq_init);block/elevator-fq.h new file +17
-0@@ -0,0 +1,473 @@ +/* +* BFQ: data structures and common functions prototypes. +* +* Based on ideas and code from CFQ: +* Copyright (C) 2003 Jens Axboe <axboe-tSWWG44O7X1aa/[email protected]> +* +* Copyright (C) 2008 Fabio Checconi <[email protected]> +* Paolo Valente <[email protected]> +* Copyright (C) 2009 Vivek Goyal <[email protected]> +* Nauman Rafique <[email protected]> +*/ + +#include <linux/blkdev.h> + +#ifndef _BFQ_SCHED_H +#define _BFQ_SCHED_H +
Quoted 9 lines
block/elevator-fq.h line 22
Quoted 5 lines
block/elevator-fq.h new file +5
-0+#define IO_IOPRIO_CLASSES 3 + +typedef u64 bfq_timestamp_t ; +typedef unsigned long bfq_weight_t ; +typedef unsigned long bfq_service_t ;Does this abstraction really provide any benefit? Why not directly use the standard C types, make the code easier to read.
I have no strong opinions on that, during debugging it helped a lot to identify the role of variables in the code, but common practice in the kernel is avoiding typedefs, so they can go now.
Quoted 66 lines
block/elevator-fq.h line 32
Quoted 10 lines
block/elevator-fq.h new file +10
-0+struct io_entity ; +struct io_queue ; + +#ifdef CONFIG_ELV_FAIR_QUEUING + +#define ELV_ATTR (name) \ + __ATTR(name, S_IRUGO|S_IWUSR, elv_##name##_show, elv_##name##_store) + +/** +* struct bfq_service_tree - per ioprio_class service tree. Comment is old, does not reflect the newer name
Quoted 15 lines
block/elevator-fq.h new file +15
-0+* @active: tree for active entities (i.e., those backlogged). +* @idle: tree for idle entities (i.e., those not backlogged, with V <= F_i). +* @first_idle: idle entity with minimum F_i. +* @last_idle: idle entity with maximum F_i. +* @vtime: scheduler virtual time. +* @wsum: scheduler weight sum; active and idle entities contribute to it. +* +* Each service tree represents a B-WF2Q+ scheduler on its own. Each +* ioprio_class has its own independent scheduler, and so its own +* bfq_service_tree. All the fields are protected by the queue lock +* of the containing efqd. +*/ +struct io_service_tree { +struct rb_root active; +struct rb_root idle;block/elevator-fq.h line 57
Quoted 10 lines
block/elevator-fq.h new file +10
-0+ +struct io_entity *first_idle; +struct io_entity *last_idle; + +bfq_timestamp_t vtime; +bfq_weight_t wsum; +}; + +/** +* struct bfq_sched_data - multi-class scheduler. Again the naming convention is broken, you need to change several bfq's to io's :)
Quoted 24 lines
block/elevator-fq.h new file +24
-0+* @active_entity: entity under service. +* @next_active: head-of-the-line entity in the scheduler. +* @service_tree: array of service trees, one per ioprio_class. +* +* bfq_sched_data is the basic scheduler queue. It supports three +* ioprio_classes, and can be used either as a toplevel queue or as +* an intermediate queue on a hierarchical setup. +* @next_active points to the active entity of the sched_data service +* trees that will be scheduled next. +* +* The supported ioprio_classes are the same as in CFQ, in descending +* priority order, IOPRIO_CLASS_RT, IOPRIO_CLASS_BE, IOPRIO_CLASS_IDLE. +* Requests from higher priority queues are served before all the +* requests from lower priority queues; among requests of the same +* queue requests are served according to B-WF2Q+. +* All the fields are protected by the queue lock of the containing bfqd. +*/ +struct io_sched_data { +struct io_entity *active_entity; +struct io_service_tree service_tree[IO_IOPRIO_CLASSES ]; +}; + +/** +* struct bfq_entity - schedulable entity.
Quoted 9 lines
block/elevator-fq.h line 86
Quoted 5 lines
block/elevator-fq.h new file +5
-0+* @rb_node: service_tree member. +* @on_st: flag, true if the entity is on a tree (either the active or +* the idle one of its service_tree). +* @finish: B-WF2Q+ finish timestamp (aka F_i). +* @start: B-WF2Q+ start timestamp (aka S_i). Could you mention what key is used in the
rb_tree? start, finish sounds like a range, so my suspicion is that start is used.
finish is used as the key, and min_start keeps the minimum ->start for the subtree rooted at the given entity (as said in the comment below).
Quoted 7 lines
block/elevator-fq.h line 89
Quoted 3 lines
block/elevator-fq.h new file +3
-0+* @tree: tree the entity is enqueued into; %NULL if not on a tree. +* @min_start: minimum start time of the (active) subtree rooted at +* this entity; used for O(log N) lookups into active trees. Used for O(log N) makes no sense to me, RBTree has a worst case lookup time of O(log N), but what is the comment saying?
it's badly written (my fault), but it intended to say that this field is used to allow the lookups to be done in O(log N). without augmenting the RB tree with min_start, lookups could not be done in O(log N), because we want a constrained minimum search.
Quoted 64 lines
Quoted 64 lines
block/elevator-fq.h new file +64
-0+* @service: service received during the last round of service. +* @budget: budget used to calculate F_i; F_i = S_i + @budget / @weight. +* @weight: weight of the queue, calculated as IOPRIO_BE_NR - @ioprio. +* @parent: parent entity, for hierarchical scheduling. +* @my_sched_data: for non-leaf nodes in the cgroup hierarchy, the +* associated scheduler queue, %NULL on leaf nodes. +* @sched_data: the scheduler queue this entity belongs to. +* @ioprio: the ioprio in use. +* @new_ioprio: when an ioprio change is requested, the new ioprio value +* @ioprio_class: the ioprio_class in use. +* @new_ioprio_class: when an ioprio_class change is requested, the new +* ioprio_class value. +* @ioprio_changed: flag, true when the user requested an ioprio or +* ioprio_class change. +* +* A bfq_entity is used to represent either a bfq_queue (leaf node in the +* cgroup hierarchy) or a bfq_group into the upper level scheduler. Each +* entity belongs to the sched_data of the parent group in the cgroup +* hierarchy. Non-leaf entities have also their own sched_data, stored +* in @my_sched_data. +* +* Each entity stores independently its priority values; this would allow +* different weights on different devices, but this functionality is not +* exported to userspace by now. Priorities are updated lazily, first +* storing the new values into the new_* fields, then setting the +* @ioprio_changed flag. As soon as there is a transition in the entity +* state that allows the priority update to take place the effective and +* the requested priority values are synchronized. +* +* The weight value is calculated from the ioprio to export the same +* interface as CFQ. When dealing with ``well-behaved'' queues (i.e., +* queues that do not spend too much time to consume their budget and +* have true sequential behavior, and when there are no external factors +* breaking anticipation) the relative weights at each level of the +* cgroups hierarchy should be guaranteed. +* All the fields are protected by the queue lock of the containing bfqd. +*/ +struct io_entity { +struct rb_node rb_node; + +int on_st; + +bfq_timestamp_t finish; +bfq_timestamp_t start; + +struct rb_root *tree; + +bfq_timestamp_t min_start; + +bfq_service_t service, budget; +bfq_weight_t weight; + +struct io_entity *parent; + +struct io_sched_data *my_sched_data; +struct io_sched_data *sched_data; + +unsigned short ioprio, new_ioprio; +unsigned short ioprio_class, new_ioprio_class; + +int ioprio_changed; +}; + +/*
Quoted 9 lines
block/elevator-fq.h line 158
Quoted 5 lines
block/elevator-fq.h new file +5
-0+* A common structure embedded by every io scheduler into their respective +* queue structure. +*/ +struct io_queue { +struct io_entity entity;So the
io_queuehas an abstract entity calledio_entitythat contains it QoS parameters? Correct?
yes
Quoted 9 lines
block/elevator-fq.h line 164
Quoted 6 lines
block/elevator-fq.h new file +6
-0+atomic_t ref; +unsigned int flags; + +/* Pointer to generic elevator data structure */ +struct elv_fq_data *efqd; +pid_t pid;Why do we store the pid?
originally it was for logging purposes
Quoted 570 lines
Quoted 14 lines
block/elevator-fq.h new file +14
-0+ +/* Number of requests queued on this io queue */ +unsigned long nr_queued; + +/* Requests dispatched from this queue */ +int dispatched; + +/* Keep a track of think time of processes in this queue */ +unsigned long last_end_request; +unsigned long ttime_total; +unsigned long ttime_samples; +unsigned long ttime_mean; + +unsigned long slice_end;block/elevator-fq.h line 188
Quoted 10 lines
block/elevator-fq.h new file +10
-0+ +/* Pointer to io scheduler's queue */ +void *sched_queue; +}; + +struct io_group { +struct io_sched_data sched_data; + +/* async_queue and idle_queue are used only for cfq */ +struct io_queue *async_queue[2 ][IOPRIO_BE_NR ];Again the 2 is confusing
block/elevator-fq.h line 198
Quoted 10 lines
block/elevator-fq.h new file +10
-0+struct io_queue *async_idle_queue; + +/* +* Used to track any pending rt requests so we can pre-empt current +* non-RT cfqq in service when this value is non-zero. +*/ +unsigned int busy_rt_queues; +}; + +struct elv_fq_data {What does fq stand for?
Quoted 5 lines
block/elevator-fq.h new file +5
-0+struct io_group *root_group; + +struct request_queue *queue; +unsigned int busy_queues; +block/elevator-fq.h line 213
Quoted 10 lines
block/elevator-fq.h new file +10
-0+/* Number of requests queued */ +int rq_queued; + +/* Pointer to the ioscheduler queue being served */ +void *active_queue; + +int rq_in_driver; +int hw_tag; +int hw_tag_samples; +int rq_in_driver_peak;Some comments of
_in_driverand_in_driver_peakwould be nice.Quoted 4 lines
block/elevator-fq.h new file +4
-0+ +/* +* elevator fair queuing layer has the capability to provide idling +* for ensuring fairness for processes doing dependent reads. block/elevator-fq.h line 227
Quoted 10 lines
block/elevator-fq.h new file +10
-0+* This might be needed to ensure fairness among two processes doing +* synchronous reads in two different cgroups. noop and deadline don't +* have any notion of anticipation/idling. As of now, these are the +* users of this functionality. +*/ +unsigned int elv_slice_idle; +struct timer_list idle_slice_timer; +struct work_struct unplug_work; + +unsigned int elv_slice[2 ];Why [2] makes the code hearder to read
Quoted 492 lines
block/elevator-fq.h new file +246
-0+}; + +extern int elv_slice_idle; +extern int elv_slice_async; + +/* Logging facilities. */ +#define elv_log_ioq(efqd, ioq, fmt, args...) \ + blk_add_trace_msg((efqd)->queue, "elv%d%c " fmt, (ioq)->pid, \ + elv_ioq_sync(ioq) ? 'S' : 'A', ##args) + +#define elv_log(efqd, fmt, args...) \ + blk_add_trace_msg((efqd)->queue, "elv " fmt, ##args) + +#define ioq_sample_valid (samples) ((samples) > 80) + +/* Some shared queue flag manipulation functions among elevators */ + +enum elv_queue_state_flags { + ELV_QUEUE_FLAG_busy =0 ,/* has requests or is under service */ + ELV_QUEUE_FLAG_sync,/* synchronous queue */ + ELV_QUEUE_FLAG_idle_window,/* elevator slice idling enabled */ + ELV_QUEUE_FLAG_wait_request,/* waiting for a request */ + ELV_QUEUE_FLAG_must_dispatch,/* must be allowed a dispatch */ + ELV_QUEUE_FLAG_slice_new,/* no requests dispatched in slice */ +ELV_QUEUE_FLAG_NR , +}; + +#define ELV_IO_QUEUE_FLAG_FNS (name) \ +static inline void elv_mark_ioq_##name(struct io_queue *ioq) \ +{ \ + (ioq)->flags |= (1 << ELV_QUEUE_FLAG_##name); \ +} \ +static inline void elv_clear_ioq_##name(struct io_queue *ioq) \ +{ \ + (ioq)->flags &= ~(1 << ELV_QUEUE_FLAG_##name); \ +} \ +static inline int elv_ioq_##name(struct io_queue *ioq) \ +{ \ + return ((ioq)->flags & (1 << ELV_QUEUE_FLAG_##name)) != 0; \ +} + +ELV_IO_QUEUE_FLAG_FNS (busy) +ELV_IO_QUEUE_FLAG_FNS (sync) +ELV_IO_QUEUE_FLAG_FNS (wait_request) +ELV_IO_QUEUE_FLAG_FNS(must_dispatch) +ELV_IO_QUEUE_FLAG_FNS(idle_window) +ELV_IO_QUEUE_FLAG_FNS(slice_new) + +static inline struct io_service_tree * +io_entity_service_tree(struct io_entity *entity) +{ + struct io_sched_data *sched_data = entity->sched_data; +unsigned int idx = entity->ioprio_class -1 ; + + BUG_ON(idx >= IO_IOPRIO_CLASSES); + BUG_ON(sched_data == NULL); + + return sched_data->service_tree + idx; +} + +/* A request got dispatched from the io_queue. Do the accounting. */ +static inline void elv_ioq_request_dispatched(struct io_queue *ioq) +{ + ioq->dispatched++; +} + +static inline int elv_ioq_slice_used (struct io_queue *ioq) +{ +if (elv_ioq_slice_new (ioq)) +return 0 ; +if (time_before (jiffies, ioq->slice_end)) +return 0 ; + +return 1 ; +} + +/* How many request are currently dispatched from the queue */ +static inline int elv_ioq_nr_dispatched (struct io_queue *ioq) +{ +return ioq->dispatched; +} + +/* How many request are currently queued in the queue */ +static inline int elv_ioq_nr_queued (struct io_queue *ioq) +{ +return ioq->nr_queued; +} + +static inline void elv_get_ioq (struct io_queue *ioq) +{ +atomic_inc (&ioq->ref); +} + +static inline void elv_ioq_set_slice_end (struct io_queue *ioq, +unsigned long slice_end) +{ + ioq->slice_end = slice_end; +} + +static inline int elv_ioq_class_idle (struct io_queue *ioq) +{ +return ioq->entity.ioprio_class ==IOPRIO_CLASS_IDLE ; +} + +static inline int elv_ioq_class_rt (struct io_queue *ioq) +{ +return ioq->entity.ioprio_class ==IOPRIO_CLASS_RT ; +} + +static inline int elv_ioq_ioprio_class (struct io_queue *ioq) +{ +return ioq->entity.new_ioprio_class; +} + +static inline int elv_ioq_ioprio (struct io_queue *ioq) +{ +return ioq->entity.new_ioprio; +} + +static inline void elv_ioq_set_ioprio_class (struct io_queue *ioq, +int ioprio_class) +{ + ioq->entity.new_ioprio_class = ioprio_class; + ioq->entity.ioprio_changed =1 ; +} + +static inline void elv_ioq_set_ioprio (struct io_queue *ioq,int ioprio) +{ + ioq->entity.new_ioprio = ioprio; + ioq->entity.ioprio_changed =1 ; +} + +static inline void *ioq_sched_queue (struct io_queue *ioq) +{ +if (ioq) +return ioq->sched_queue; +return NULL ; +} + +static inline struct io_group *ioq_to_io_group (struct io_queue *ioq) +{ + return container_of(ioq->entity.sched_data, struct io_group, + sched_data); +} + +extern ssize_t elv_slice_idle_show (struct elevator_queue *q,char *name); +extern ssize_t elv_slice_idle_store (struct elevator_queue *q,const char *name, +size_t count); +extern ssize_t elv_slice_sync_show (struct elevator_queue *q,char *name); +extern ssize_t elv_slice_sync_store (struct elevator_queue *q,const char *name, +size_t count); +extern ssize_t elv_slice_async_show (struct elevator_queue *q,char *name); +extern ssize_t elv_slice_async_store (struct elevator_queue *q,const char *name, +size_t count); + +/* Functions used by elevator.c */ +extern int elv_init_fq_data (struct request_queue *q,struct elevator_queue *e); +extern void elv_exit_fq_data (struct elevator_queue *e); +extern void elv_exit_fq_data_post (struct elevator_queue *e); + +extern void elv_ioq_request_add (struct request_queue *q,struct request *rq); +extern void elv_ioq_request_removed (struct elevator_queue *e, +struct request *rq); +extern void elv_fq_dispatched_request (struct elevator_queue *e, +struct request *rq); + +extern void elv_fq_activate_rq (struct request_queue *q,struct request *rq); +extern void elv_fq_deactivate_rq (struct request_queue *q,struct request *rq); + +extern void elv_ioq_completed_request (struct request_queue *q, +struct request *rq); + +extern void *elv_fq_select_ioq (struct request_queue *q,int force); +extern struct io_queue *rq_ioq (struct request *rq); + +/* Functions used by io schedulers */ +extern void elv_put_ioq (struct io_queue *ioq); +extern void __elv_ioq_slice_expired (struct request_queue *q, +struct io_queue *ioq); +extern int elv_init_ioq (struct elevator_queue *eq,struct io_queue *ioq, +void *sched_queue,int ioprio_class,int ioprio,int is_sync); +extern void elv_schedule_dispatch (struct request_queue *q); +extern int elv_hw_tag (struct elevator_queue *e); +extern void *elv_active_sched_queue (struct elevator_queue *e); +extern int elv_mod_idle_slice_timer (struct elevator_queue *eq, +unsigned long expires); +extern int elv_del_idle_slice_timer (struct elevator_queue *eq); +extern unsigned int elv_get_slice_idle (struct elevator_queue *eq); +extern void *io_group_async_queue_prio (struct io_group *iog,int ioprio_class, +int ioprio); +extern void io_group_set_async_queue (struct io_group *iog,int ioprio_class, +int ioprio,struct io_queue *ioq); +extern struct io_group *io_lookup_io_group_current (struct request_queue *q); +extern int elv_nr_busy_ioq (struct elevator_queue *e); +extern struct io_queue *elv_alloc_ioq (struct request_queue *q,gfp_t gfp_mask); +extern void elv_free_ioq (struct io_queue *ioq); + +#else /* CONFIG_ELV_FAIR_QUEUING */ + +static inline int elv_init_fq_data (struct request_queue *q, +struct elevator_queue *e) +{ +return 0 ; +} + +static inline void elv_exit_fq_data (struct elevator_queue *e) {} +static inline void elv_exit_fq_data_post (struct elevator_queue *e) {} + +static inline void elv_fq_activate_rq (struct request_queue *q, +struct request *rq) +{ +} + +static inline void elv_fq_deactivate_rq (struct request_queue *q, +struct request *rq) +{ +} + +static inline void elv_fq_dispatched_request (struct elevator_queue *e, +struct request *rq) +{ +} + +static inline void elv_ioq_request_removed (struct elevator_queue *e, +struct request *rq) +{ +} + +static inline void elv_ioq_request_add (struct request_queue *q, +struct request *rq) +{ +} + +static inline void elv_ioq_completed_request (struct request_queue *q, +struct request *rq) +{ +} + +static inline void *ioq_sched_queue (struct io_queue *ioq) {return NULL ; } +static inline struct io_queue *rq_ioq (struct request *rq) {return NULL ; } +static inline void *elv_fq_select_ioq (struct request_queue *q,int force) +{ +return NULL ; +} +#endif /* CONFIG_ELV_FAIR_QUEUING */ +#endif /* _BFQ_SCHED_H */ block/elevator.c +35
-11@@ -231,6 +231,9 @@ static struct elevator_queue *elevator_alloc(struct request_queue *q,for (i =0 ; i <ELV_HASH_ENTRIES ; i++)INIT_HLIST_HEAD (&eq->hash[i]); +if (elv_init_fq_data (q, eq)) + goto err; +return eq; err:kfree (eq);@@ -301,9 +304,11 @@ EXPORT_SYMBOL(elevator_init);void elevator_exit (struct elevator_queue *e) {mutex_lock (&e->sysfs_lock); +elv_exit_fq_data (e);if (e->ops->elevator_exit_fn) e->ops->elevator_exit_fn (e); e->ops =NULL ; +elv_exit_fq_data_post (e);mutex_unlock (&e->sysfs_lock);kobject_put (&e->kobj);@@ -314,6 +319,8 @@ static void elv_activate_rq(struct request_queue *q, struct request *rq) {struct elevator_queue *e = q->elevator; +elv_fq_activate_rq (q, rq); +if (e->ops->elevator_activate_req_fn) e->ops->elevator_activate_req_fn (q, rq); }@@ -322,6 +329,8 @@ static void elv_deactivate_rq(struct request_queue *q, struct request *rq) {struct elevator_queue *e = q->elevator; +elv_fq_deactivate_rq (q, rq); +if (e->ops->elevator_deactivate_req_fn) e->ops->elevator_deactivate_req_fn (q, rq); }@@ -446,6 +455,7 @@ void elv_dispatch_sort(struct request_queue *q, struct request *rq)elv_rqhash_del (q, rq); q->nr_sorted--; +elv_fq_dispatched_request (q->elevator, rq); boundary = q->end_sector; stop_flags =REQ_SOFTBARRIER |REQ_HARDBARRIER |REQ_STARTED ;@@ -486,6 +496,7 @@ void elv_dispatch_add_tail(struct request_queue *q, struct request *rq)elv_rqhash_del (q, rq); q->nr_sorted--; +elv_fq_dispatched_request (q->elevator, rq); q->end_sector =rq_end_sector (rq); q->boundary_rq = rq;@@ -553,6 +564,7 @@ void elv_merge_requests(struct request_queue *q, struct request *rq,elv_rqhash_del (q, next); q->nr_sorted--; +elv_ioq_request_removed (e, next); q->last_merge = rq; }@@ -657,12 +669,8 @@ void elv_insert(struct request_queue *q, struct request *rq, int where) q->last_merge = rq; } -/* -* Some ioscheds (cfq) run q->request_fn directly, so -* rq cannot be accessed after calling -* elevator_add_req_fn. -*/ q->elevator->ops->elevator_add_req_fn (q, rq); +elv_ioq_request_add (q, rq);break ;case ELEVATOR_INSERT_REQUEUE :@@ -872,13 +880,12 @@ void elv_dequeue_request(struct request_queue *q, struct request *rq)int elv_queue_empty (struct request_queue *q) { -struct elevator_queue *e = q->elevator; -if (!list_empty (&q->queue_head))return 0 ; -if (e->ops->elevator_queue_empty_fn) -return e->ops->elevator_queue_empty_fn (q); +/* Hopefully nr_sorted works and no need to call queue_empty_fn */ +if (q->nr_sorted) +return 0 ;return 1 ; }@@ -953,8 +960,11 @@ void elv_completed_request(struct request_queue *q, struct request *rq)*/ if (blk_account_rq (rq)) { q->in_flight--; -if (blk_sorted_rq (rq) && e->ops->elevator_completed_req_fn) - e->ops->elevator_completed_req_fn (q, rq); +if (blk_sorted_rq (rq)) { +if (e->ops->elevator_completed_req_fn) + e->ops->elevator_completed_req_fn (q, rq); +elv_ioq_completed_request (q, rq); + } }/* @@ -1242,3 +1252,17 @@ struct request *elv_rb_latter_request(struct request_queue *q,return NULL ; }EXPORT_SYMBOL (elv_rb_latter_request); + +/* Get the io scheduler queue pointer. For cfq, it is stored in rq->ioq*/ +void *elv_get_sched_queue (struct request_queue *q,struct request *rq) +{ +return ioq_sched_queue (rq_ioq (rq)); +} +EXPORT_SYMBOL (elv_get_sched_queue); + +/* Select an ioscheduler queue to dispatch request from. */ +void *elv_select_sched_queue (struct request_queue *q,int force) +{ +return ioq_sched_queue (elv_fq_select_ioq (q, force)); +} +EXPORT_SYMBOL (elv_select_sched_queue);include/linux/blkdev.h +5
-0@@ -245,6 +245,11 @@ struct request {/* for bidi */ struct request *next_rq; + +#ifdef CONFIG_ELV_FAIR_QUEUING +/* io queue request belongs to */ +struct io_queue *ioq; +#endif };static inline unsigned short req_get_ioprio (struct request *req)include/linux/elevator.h +51
-0@@ -2,6 +2,7 @@#define _LINUX_ELEVATOR_H#include <linux/percpu.h> +#include "../../block/elevator-fq.h" #ifdef CONFIG_BLOCK @@ -29,6 +30,18 @@ typedef void (elevator_deactivate_req_fn) (struct request_queue *, struct requestypedef void *(elevator_init_fn ) (struct request_queue *);typedef void (elevator_exit_fn ) (struct elevator_queue *); +#ifdef CONFIG_ELV_FAIR_QUEUING +typedef void (elevator_free_sched_queue_fn ) (struct elevator_queue *,void *); +typedef void (elevator_active_ioq_set_fn ) (struct request_queue *,void *,int ); +typedef void (elevator_active_ioq_reset_fn ) (struct request_queue *,void *); +typedef void (elevator_arm_slice_timer_fn ) (struct request_queue *,void *); +typedef int (elevator_should_preempt_fn ) (struct request_queue *,void *, +struct request *); +typedef int (elevator_update_idle_window_fn ) (struct elevator_queue *,void *, +struct request *); +typedef struct io_queue * (elevator_close_cooperator_fn ) (struct request_queue *, +void *,int probe); +#endif struct elevator_ops {@@ -56,6 +69,17 @@ struct elevator_opselevator_init_fn *elevator_init_fn;elevator_exit_fn *elevator_exit_fn;void (*trim)(struct io_context *); + +#ifdef CONFIG_ELV_FAIR_QUEUING +elevator_free_sched_queue_fn *elevator_free_sched_queue_fn; +elevator_active_ioq_set_fn *elevator_active_ioq_set_fn; +elevator_active_ioq_reset_fn *elevator_active_ioq_reset_fn; + +elevator_arm_slice_timer_fn *elevator_arm_slice_timer_fn; +elevator_should_preempt_fn *elevator_should_preempt_fn; +elevator_update_idle_window_fn *elevator_update_idle_window_fn; +elevator_close_cooperator_fn *elevator_close_cooperator_fn; +#endif };#define ELV_NAME_MAX (16)@@ -76,6 +100,9 @@ struct elevator_typestruct elv_fs_entry *elevator_attrs;char elevator_name[ELV_NAME_MAX ];struct module *elevator_owner; +#ifdef CONFIG_ELV_FAIR_QUEUING +int elevator_features; +#endif };/* @@ -89,6 +116,10 @@ struct elevator_queuestruct elevator_type *elevator_type;struct mutex sysfs_lock;struct hlist_head *hash; +#ifdef CONFIG_ELV_FAIR_QUEUING +/* fair queuing data */ +struct elv_fq_data efqd; +#endif };/* @@ -209,5 +240,25 @@ enum { __val; \ }) +/* iosched can let elevator know their feature set/capability */ +#ifdef CONFIG_ELV_FAIR_QUEUING + +/* iosched wants to use fq logic of elevator layer */ +#define ELV_IOSCHED_NEED_FQ 1 + +static inline int elv_iosched_fair_queuing_enabled (struct elevator_queue *e) +{ +return (e->elevator_type->elevator_features) &ELV_IOSCHED_NEED_FQ ; +} + +#else /* ELV_IOSCHED_FAIR_QUEUING */ + +static inline int elv_iosched_fair_queuing_enabled (struct elevator_queue *e) +{ +return 0 ; +} +#endif /* ELV_IOSCHED_FAIR_QUEUING */ +extern void *elv_get_sched_queue (struct request_queue *q,struct request *rq); +extern void *elv_select_sched_queue (struct request_queue *q,int force);#endif /* CONFIG_BLOCK */ #endif Signature · 1 line
1.6.0.6Signature · 1 line
Balbir
Re: [PATCH 02/20] io-controller: Common flat fair queuing code in elevaotor layer
In reply to a message not in the archiveQuoted 14 lines · Balbir Singh
Quoted 11 lines · Vivek Goyal
This is common fair queuing code in elevator layer. This is controlled by config option
CONFIG_ELV_FAIR_QUEUING. This patch initially only introduces flat fair queuing support where there is only one group, "root group" and all the tasks belong to root group.This elevator layer changes are backward compatible. That means any ioscheduler using old interfaces will continue to work.
This code is essentially the CFQ code for fair queuing. The primary difference is that flat rounding robin algorithm of CFQ has been replaced with BFQ (WF2Q+).
Quoted 6 lines
The patch is quite long and to be honest requires a long time to review, which I don't mind. I suspect my frequently diverted mind is likely to miss a lot in a big patch like this. Could you consider splitting this further if possible. I think you'll notice the number of reviews will also increase.
Hi Balbir,
Thanks for the review. Yes, this is a big patch. I will try to break it down further.
Fabio has already responded to most of the questions. I will try to cover rest.
[..]
Quoted 10 lines
block/elevator-fq.c line 35
Quoted 7 lines
block/elevator-fq.c new file +7
-0+static inline struct io_queue *elv_close_cooperator (struct request_queue *q, +struct io_queue *ioq,int probe); +struct io_entity *bfq_lookup_next_entity (struct io_sched_data *sd, +int extract); + +static inline int elv_prio_slice (struct elv_fq_data *efqd,int sync, +unsigned short prio)Why is the return type int and not unsigned int or unsigned long? Can the return value ever be negative?
Actually this function was a replacement for cfq_prio_slice() hence return type int. But as slice value can never be negative, I can make it unsigned int.
[..]
Quoted 6 lines
Quoted 6 lines
+ *
bfq_gt- compare two timestamps.
+ * @a: first ts.
+ * @b: second ts.
+ *
+ * Return @a > @b, dealing with wrapping correctly.
+ */
Quoted 8 lines
block/elevator-fq.c line 71
Quoted 5 lines
block/elevator-fq.c new file +5
-0+static inline int bfq_gt (bfq_timestamp_t a,bfq_timestamp_t b) +{ +return (s64)(a - b) >0 ; +} +a and b are of type u64, but cast to s64 to deal with wrapping? Correct?
Yes.
Quoted 15 lines
block/elevator-fq.c line 81
Quoted 11 lines
block/elevator-fq.c new file +10
-0+/** +* bfq_delta - map service into the virtual time domain. +* @service: amount of service. +* @weight: scale factor. +*/ +static inline bfq_timestamp_t bfq_delta (bfq_service_t service, +bfq_weight_t weight) +{ +bfq_timestamp_t d = (bfq_timestamp_t )service <<WFQ_SERVICE_SHIFT ; +Why is the case required? Does the compiler complain? service is already of the correct type.
Quoted 6 lines
block/elevator-fq.c line 82
Quoted 1 line
block/elevator-fq.c new file +1
-0+do_div (d, weight);On a 64 system both d and weight are 64 bit, but on a 32 bit system weight is 32 bits.
do_divexpects a 64 bit dividend and 32 bit divisor
- no?
d is of type "bfq_timestamp_t" which is u64 irrespective of 64 or 32 bit system. I think it might make sense to change type of "weight" from unsigned long to unsigned int so that it is 32bit on both 64 and 32bit systems. Will do...
Quoted 10 lines
Quoted 10 lines
block/elevator-fq.c new file +10
-0+return d; +} + +/** +* bfq_calc_finish - assign the finish time to an entity. +* @entity: the entity to act upon. +* @service: the service to be charged to the entity. +*/ +static inline void bfq_calc_finish (struct io_entity *entity, +bfq_service_t service)
Quoted 9 lines
block/elevator-fq.c line 97
Quoted 5 lines
block/elevator-fq.c new file +5
-0+{ +BUG_ON (entity->weight ==0 ); + + entity->finish = entity->start +bfq_delta (service, entity->weight); +}Should we
BUG_ON(entity->finish==entity->start)? Or is that expected when the entity has no service time left.
As Fabio said, that with preemption logic, I guess theoritically, it is possible that a io queue is preempted without any service received and requeued back. Hence it might not be a very good idea to BUG_ON(entity->finish == entity->start);
[..]
Quoted 10 lines
block/elevator-fq.c line 134
Quoted 7 lines
block/elevator-fq.c new file +7
-0+/** +* bfq_extract - remove an entity from a tree. +* @root: the tree root. +* @entity: the entity to remove. +*/ +static inline void bfq_extract (struct rb_root *root,struct io_entity *entity) +{Extract is not common terminology, why not use
bfq_remove()?
*_remove() also sounds good. Will replace *_extract() with *_remove().
Quoted 6 lines
block/elevator-fq.c line 138
Quoted 4 lines
block/elevator-fq.c new file +4
-0+BUG_ON (entity->tree != root); + + entity->tree =NULL ; +rb_erase (&entity->rb_node, root);Don't you want to make
entity->tree= NULL afterrb_erase?
As Fabio said that it happens under queue spinlock held. But from readability point of view, it probably looks better to first remove it from rb tree then reset entity fields. Will change the order...
Quoted 28 lines
Quoted 7 lines
block/elevator-fq.c new file +6
-0+} + +/** +* bfq_idle_extract - extract an entity from the idle tree. +* @st: the service tree of the owning @entity. +* @entity: the entity being removed. block/elevator-fq.c line 154
Quoted 10 lines
block/elevator-fq.c new file +10
-0+*/ +static void bfq_idle_extract (struct io_service_tree *st, +struct io_entity *entity) +{ +struct rb_node *next; + +BUG_ON (entity->tree != &st->idle); + +if (entity == st->first_idle) { + next =rb_next (&entity->rb_node);What happens if next is NULL?
block/elevator-fq.c line 159
Quoted 5 lines
block/elevator-fq.c new file +5
-0+ st->first_idle =bfq_entity_of (next); + } + +if (entity == st->last_idle) { + next =rb_prev (&entity->rb_node);What happens if next is NULL?
Quoted 1 line
block/elevator-fq.c line 160
Quoted 1 line
block/elevator-fq.c new file +1
-0+ st->last_idle =bfq_entity_of (next);
bfq_entity_of() is capable of handling next == NULL.
I can change it to following if you think it is more readable.
if (entity == st->first_idle) {
next = rb_next(&entity->rb_node);
if (next)
st->first_idle = bfq_entity_of(next);
else
st->first_idle = NULL;
}[..]
Quoted 5 lines
Quoted 5 lines
block/elevator-fq.c new file +5
-0+static void elv_ioq_update_io_thinktime (struct io_queue *ioq) +{ +struct elv_fq_data *efqd = ioq->efqd; +unsigned long elapsed = jiffies - ioq->last_end_request; +unsigned long ttime =min (elapsed,2UL * efqd->elv_slice_idle);
Quoted 8 lines
block/elevator-fq.c line 962
Quoted 5 lines
block/elevator-fq.c new file +5
-0+ + ioq->ttime_samples = (7 *ioq->ttime_samples +256 ) /8 ; + ioq->ttime_total = (7 *ioq->ttime_total +256 *ttime) /8 ; + ioq->ttime_mean = (ioq->ttime_total +128 ) / ioq->ttime_samples; +}Not sure I understand the magical 7, 8, 2, 128 and 256. Please help me understand the algorithm.
Taken from CFQ.
Quoted 8 lines
Quoted 8 lines
block/elevator-fq.c new file +8
-0+int elv_init_ioq (struct elevator_queue *eq,struct io_queue *ioq, +void *sched_queue,int ioprio_class,int ioprio, +int is_sync) +{ +struct elv_fq_data *efqd = &eq->efqd; +struct io_group *iog =io_lookup_io_group_current (efqd->queue); + +RB_CLEAR_NODE (&ioq->entity.rb_node);
Quoted 9 lines
block/elevator-fq.c line 1040
Quoted 5 lines
block/elevator-fq.c new file +5
-0+atomic_set (&ioq->ref,0 ); + ioq->efqd = efqd; +elv_ioq_set_ioprio_class (ioq, ioprio_class); +elv_ioq_set_ioprio (ioq, ioprio); + ioq->pid = current->pid;Is pid used for cgroup association later? I don't see why we save the pid otherwise? If yes, why not store the cgroup of the
current->pid?
This is just for logging purposes (blktrace), useful for CFQ where every task context sets up one queue and this number becomes the identifier for the queue. Look at elv_log_ioq(), which uses ioq->pid.
[..]
Quoted 3 lines
Quoted 1 line
+ * coop tells that io scheduler selected a queue for us and we did not
coop?
coop refers to "cooperating". I guess "coop" is not descriptive. I will change the name to "cooperating" and also put more description for clarity.
[..]
Quoted 23 lines
Quoted 23 lines
block/elevator-fq.h new file +17
-0@@ -0,0 +1,473 @@ +/* +* BFQ: data structures and common functions prototypes. +* +* Based on ideas and code from CFQ: +* Copyright (C) 2003 Jens Axboe <axboe-tSWWG44O7X1aa/[email protected]> +* +* Copyright (C) 2008 Fabio Checconi <[email protected]> +* Paolo Valente <[email protected]> +* Copyright (C) 2009 Vivek Goyal <[email protected]> +* Nauman Rafique <[email protected]> +*/ + +#include <linux/blkdev.h> + +#ifndef _BFQ_SCHED_H +#define _BFQ_SCHED_H +
Quoted 8 lines
block/elevator-fq.h line 22
Quoted 5 lines
block/elevator-fq.h new file +5
-0+#define IO_IOPRIO_CLASSES 3 + +typedef u64 bfq_timestamp_t ; +typedef unsigned long bfq_weight_t ; +typedef unsigned long bfq_service_t ;Does this abstraction really provide any benefit? Why not directly use the standard C types, make the code easier to read.
I think using standard C type is better now. Will get rid of these abstractions. Fabio also seems to be ok with this change.
Quoted 13 lines
block/elevator-fq.h line 32
Quoted 11 lines
block/elevator-fq.h new file +10
-0+struct io_entity ; +struct io_queue ; + +#ifdef CONFIG_ELV_FAIR_QUEUING + +#define ELV_ATTR (name) \ + __ATTR(name, S_IRUGO|S_IWUSR, elv_##name##_show, elv_##name##_store) + +/** +* struct bfq_service_tree - per ioprio_class service tree. Comment is old, does not reflect the newer name
Yes, this is all over the code. I have not taken care of updating the comments from original bfq code. Will do it.
Quoted 21 lines
Quoted 21 lines
block/elevator-fq.h new file +20
-0+* @active: tree for active entities (i.e., those backlogged). +* @idle: tree for idle entities (i.e., those not backlogged, with V <= F_i). +* @first_idle: idle entity with minimum F_i. +* @last_idle: idle entity with maximum F_i. +* @vtime: scheduler virtual time. +* @wsum: scheduler weight sum; active and idle entities contribute to it. +* +* Each service tree represents a B-WF2Q+ scheduler on its own. Each +* ioprio_class has its own independent scheduler, and so its own +* bfq_service_tree. All the fields are protected by the queue lock +* of the containing efqd. +*/ +struct io_service_tree { +struct rb_root active; +struct rb_root idle; + +struct io_entity *first_idle; +struct io_entity *last_idle; + +bfq_timestamp_t vtime;
Quoted 8 lines
block/elevator-fq.h line 57
Quoted 5 lines
block/elevator-fq.h new file +5
-0+bfq_weight_t wsum; +}; + +/** +* struct bfq_sched_data - multi-class scheduler. Again the naming convention is broken, you need to change several bfq's to io's :)
Yes. Will do. :-)
Quoted 10 lines
block/elevator-fq.h line 158
Quoted 6 lines
block/elevator-fq.h new file +6
-0+/* +* A common structure embedded by every io scheduler into their respective +* queue structure. +*/ +struct io_queue { +struct io_entity entity;So the
io_queuehas an abstract entity calledio_entitythat contains it QoS parameters? Correct?
Quoted 8 lines
block/elevator-fq.h line 164
Quoted 6 lines
block/elevator-fq.h new file +6
-0+atomic_t ref; +unsigned int flags; + +/* Pointer to generic elevator data structure */ +struct elv_fq_data *efqd; +pid_t pid;Why do we store the pid?
pid of the process which caused io queue creation.
Quoted 19 lines
Quoted 19 lines
block/elevator-fq.h new file +18
-0+ +/* Number of requests queued on this io queue */ +unsigned long nr_queued; + +/* Requests dispatched from this queue */ +int dispatched; + +/* Keep a track of think time of processes in this queue */ +unsigned long last_end_request; +unsigned long ttime_total; +unsigned long ttime_samples; +unsigned long ttime_mean; + +unsigned long slice_end; + +/* Pointer to io scheduler's queue */ +void *sched_queue; +};
Quoted 9 lines
block/elevator-fq.h line 188
Quoted 6 lines
block/elevator-fq.h new file +6
-0+ +struct io_group { +struct io_sched_data sched_data; + +/* async_queue and idle_queue are used only for cfq */ +struct io_queue *async_queue[2 ][IOPRIO_BE_NR ];Again the 2 is confusing
Taken from CFQ. CFQ supports 8 prio levels for RT and BE class. We maintain one async queue pointer per prio level for both RT and BE class. Above number 2 is for RT and BE class.
Quoted 4 lines
Quoted 4 lines
block/elevator-fq.h new file +4
-0+struct io_queue *async_idle_queue; + +/* +* Used to track any pending rt requests so we can pre-empt current
Quoted 8 lines
block/elevator-fq.h line 198
Quoted 6 lines
block/elevator-fq.h new file +6
-0+* non-RT cfqq in service when this value is non-zero. +*/ +unsigned int busy_rt_queues; +}; + +struct elv_fq_data {What does fq stand for?
Fair queuing. Any suggestions to make it better?
Quoted 10 lines
Quoted 10 lines
block/elevator-fq.h new file +9
-0+struct io_group *root_group; + +struct request_queue *queue; +unsigned int busy_queues; + +/* Number of requests queued */ +int rq_queued; + +/* Pointer to the ioscheduler queue being served */
Quoted 8 lines
block/elevator-fq.h line 213
Quoted 6 lines
block/elevator-fq.h new file +6
-0+void *active_queue; + +int rq_in_driver; +int hw_tag; +int hw_tag_samples; +int rq_in_driver_peak;Some comments of
_in_driverand_in_driver_peakwould be nice.
Taken from CFQ. So somebody familiar with CFQ code can quickly relate. But anyway, I will put two lines of comments.
Quoted 17 lines
Quoted 5 lines
block/elevator-fq.h new file +4
-0+ +/* +* elevator fair queuing layer has the capability to provide idling +* for ensuring fairness for processes doing dependent reads. block/elevator-fq.h line 227
Quoted 10 lines
block/elevator-fq.h new file +10
-0+* This might be needed to ensure fairness among two processes doing +* synchronous reads in two different cgroups. noop and deadline don't +* have any notion of anticipation/idling. As of now, these are the +* users of this functionality. +*/ +unsigned int elv_slice_idle; +struct timer_list idle_slice_timer; +struct work_struct unplug_work; + +unsigned int elv_slice[2 ];Why [2] makes the code hearder to read
Taken from CFQ. it represents base slice length for sync and async queues. With put a line of comment.
Thanks
Vivek