IO scheduler based IO controller V7
Subsystems (19)
PATCH 22/24 io-controller: Per io group bdi congestion interface
In reply to Vivek Goyalo So far there used to be only one pair or queue of request descriptors (one for sync and one for async) per device and number of requests allocated used to decide whether associated bdi is congested or not.
Now with per io group request descriptor infrastructure, there is a pair of request descriptor queue per io group per device. So it might happen that overall request queue is not congested but a particular io group bio belongs to is congested.
Or, it could be otherwise that group is not congested but overall queue is congested. This can happen if user has not properly set the request descriptors limits for queue and groups.
(q->nr_requests < nr_groups * q->nr_group_requests)
Hence there is a need for new interface which can query deivce congestion status per group. This group is determined by the "struct page" IO will be done for. If page is null, then group is determined from the current task context.
o This patch introduces new set of function bdi_*_congested_group(), which take "struct page" as addition argument. These functions will call the block layer and in trun elevator to find out if the io group the page will go into is congested or not.
o Currently I have introduced the core functions and migrated most of the users. But there might be still some left. This is an ongoing TODO item.
o There are some io_get_io_group() related changes which should be pushed into higher patches. Still testing this patch. Will push these changes up in next posting.
--- block/blk-core.c | 21 ++++++++++++++ block/elevator-fq.c | 62 +++++++++++++++++++++++++++++++++++++++++++ block/elevator-fq.h | 6 ++++ drivers/md/dm-table.c | 11 +++++--- drivers/md/dm.c | 7 +++-- drivers/md/dm.h | 3 +- drivers/md/linear.c | 7 +++- drivers/md/multipath.c | 7 +++- drivers/md/raid0.c | 6 +++- drivers/md/raid1.c | 9 ++++-- drivers/md/raid10.c | 6 +++- drivers/md/raid5.c | 2 +- fs/afs/write.c | 8 +++++- fs/btrfs/disk-io.c | 6 +++- fs/btrfs/extent_io.c | 12 ++++++++ fs/btrfs/volumes.c | 8 ++++- fs/cifs/file.c | 11 +++++++ fs/ext2/ialloc.c | 2 +- fs/gfs2/aops.c | 12 ++++++++ fs/nilfs2/segbuf.c | 3 +- fs/xfs/linux-2.6/xfs_aops.c | 2 +- fs/xfs/linux-2.6/xfs_buf.c | 2 +- include/linux/backing-dev.h | 61 ++++++++++++++++++++++++++++++++++++++++- include/linux/blkdev.h | 5 +++ mm/backing-dev.c | 62 +++++++++++++++++++++++++++++++++++++++++++ mm/page-writeback.c | 11 +++++++ mm/readahead.c | 2 +- 27 files changed, 322 insertions(+), 32 deletions(-)
block/blk-core.c +21 -0
@@ -90,6 +90,27 @@ void blk_queue_congestion_threshold(struct request_queue *q) q->nr_congestion_off = nr; } +#ifdef CONFIG_GROUP_IOSCHED +int blk_queue_io_group_congested (struct backing_dev_info *bdi,int bdi_bits, +struct page *page) +{ +int ret =0 ; +struct request_queue *q = bdi->unplug_io_data; + +if (!q || !q->elevator) +return bdi_congested (bdi, bdi_bits); + +/* Do we need to hold queue lock? */ +if (bdi_bits & (1 << BDI_sync_congested)) + ret |=elv_io_group_congested (q, page,1 ); + +if (bdi_bits & (1 << BDI_async_congested)) + ret |=elv_io_group_congested (q, page,0 ); + +return ret; +} +#endif +/** * blk_get_backing_dev_info - get the address of a queue's backing_dev_info * @bdev: device
block/elevator-fq.c +62 -0
86 lines not on this page · raw message
block/elevator-fq.h +6 -0
@@ -266,6 +266,10 @@ struct io_group {/* Single ioq per group, used for noop, deadline, anticipatory */ struct io_queue *ioq; +/* io group congestion on and off threshold for request descriptors */ +unsigned int nr_congestion_on; +unsigned int nr_congestion_off; +/* request list associated with the group */ struct request_list rl; };
@@ -560,6 +564,8 @@ extern struct io_queue *elv_lookup_ioq_bio(struct request_queue *q, struct bio *bio);extern struct request_list *io_group_get_request_list (struct request_queue *q,struct bio *bio); +extern int elv_io_group_congested (struct request_queue *q,struct page *page, +int sync);/* Sets the single ioq associated with the io group. (noop, deadline, AS) */ static inline void io_group_set_ioq (struct io_group *iog,struct io_queue *ioq)
drivers/md/dm-table.c +7 -4
@@ -1175,7 +1175,8 @@ int dm_table_resume_targets(struct dm_table *t)return 0 ; } -int dm_table_any_congested (struct dm_table *t,int bdi_bits) +int dm_table_any_congested (struct dm_table *t,int bdi_bits,struct page *page, +int group) {struct dm_dev_internal *dd;struct list_head *devices =dm_table_get_devices (t);
@@ -1185,9 +1186,11 @@ int dm_table_any_congested(struct dm_table *t, int bdi_bits)struct request_queue *q =bdev_get_queue (dd->dm_dev.bdev);char b[BDEVNAME_SIZE ]; -if (likely (q)) - r |=bdi_congested (&q->backing_dev_info, bdi_bits); -else +if (likely (q)) { +struct backing_dev_info *bdi = &q->backing_dev_info; + r |= group ?bdi_congested_group (bdi, bdi_bits, page) + :bdi_congested (bdi, bdi_bits); + }else DMWARN_LIMIT ("%s: any_congested: nonexistent device %s" ,dm_device_name (t->md),bdevname (dd->dm_dev.bdev, b));
drivers/md/dm.c +4 -3
@@ -1608,7 +1608,8 @@ static void dm_unplug_all(struct request_queue *q) } } -static int dm_any_congested (void *congested_data,int bdi_bits) +static int dm_any_congested (void *congested_data,int bdi_bits, +struct page *page,int group) {int r = bdi_bits;struct mapped_device *md = congested_data;
@@ -1625,8 +1626,8 @@ static int dm_any_congested(void *congested_data, int bdi_bits) r = md->queue->backing_dev_info.state & bdi_bits; else - r =dm_table_any_congested (map, bdi_bits); - + r =dm_table_any_congested (map, bdi_bits, page, + group);dm_table_put (map); } }
drivers/md/dm.h +2 -1
@@ -57,7 +57,8 @@ struct list_head *dm_table_get_devices(struct dm_table *t);void dm_table_presuspend_targets (struct dm_table *t);void dm_table_postsuspend_targets (struct dm_table *t);int dm_table_resume_targets (struct dm_table *t); -int dm_table_any_congested (struct dm_table *t,int bdi_bits); +int dm_table_any_congested (struct dm_table *t,int bdi_bits,struct page *page, +int group);int dm_table_any_busy_target (struct dm_table *t);int dm_table_set_type (struct dm_table *t);unsigned dm_table_get_type (struct dm_table *t);
drivers/md/linear.c +5 -2
@@ -102,7 +102,7 @@ static void linear_unplug(struct request_queue *q)rcu_read_unlock (); } -static int linear_congested (void *data,int bits) +static int linear_congested (void *data,int bits,struct page *page,int group) {mddev_t *mddev = data;linear_conf_t *conf;
@@ -113,7 +113,10 @@ static int linear_congested(void *data, int bits)for (i =0 ; i < mddev->raid_disks && !ret ; i++) {struct request_queue *q =bdev_get_queue (conf->disks[i].rdev->bdev); - ret |=bdi_congested (&q->backing_dev_info, bits); +struct backing_dev_info *bdi = &q->backing_dev_info; + + ret |= group ?bdi_congested_group (bdi, bits, page) : +bdi_congested (bdi, bits); }rcu_read_unlock ();
drivers/md/multipath.c +5 -2
@@ -192,7 +192,8 @@ static void multipath_status (struct seq_file *seq, mddev_t *mddev)seq_printf (seq,"]" ); } -static int multipath_congested (void *data,int bits) +static int multipath_congested (void *data,int bits,struct page *page, +int group) {mddev_t *mddev = data;multipath_conf_t *conf = mddev->private;
@@ -203,8 +204,10 @@ static int multipath_congested(void *data, int bits)mdk_rdev_t *rdev =rcu_dereference (conf->multipaths[i].rdev);if (rdev && !test_bit (Faulty, &rdev->flags)) {struct request_queue *q =bdev_get_queue (rdev->bdev); +struct backing_dev_info *bdi = &q->backing_dev_info; - ret |=bdi_congested (&q->backing_dev_info, bits); + ret |= group ?bdi_congested_group (bdi, bits, page) + :bdi_congested (bdi, bits);/* Just like multipath_map, we just check the * first available device */
drivers/md/raid0.c +4 -2
@@ -37,7 +37,7 @@ static void raid0_unplug(struct request_queue *q) } } -static int raid0_congested (void *data,int bits) +static int raid0_congested (void *data,int bits,struct page *page,int group) {mddev_t *mddev = data;raid0_conf_t *conf = mddev->private;
@@ -46,8 +46,10 @@ static int raid0_congested(void *data, int bits)for (i =0 ; i < mddev->raid_disks && !ret ; i++) {struct request_queue *q =bdev_get_queue (devlist[i]->bdev); +struct backing_dev_info *bdi = &q->backing_dev_info; - ret |=bdi_congested (&q->backing_dev_info, bits); + ret |= group ?bdi_congested_group (bdi, bits, page) + :bdi_congested (bdi, bits); }return ret; }
drivers/md/raid1.c +6 -3
@@ -570,7 +570,7 @@ static void raid1_unplug(struct request_queue *q)md_wakeup_thread (mddev->thread); } -static int raid1_congested (void *data,int bits) +static int raid1_congested (void *data,int bits,struct page *page,int group) {mddev_t *mddev = data;conf_t *conf = mddev->private;
@@ -581,14 +581,17 @@ static int raid1_congested(void *data, int bits)mdk_rdev_t *rdev =rcu_dereference (conf->mirrors[i].rdev);if (rdev && !test_bit (Faulty, &rdev->flags)) {struct request_queue *q =bdev_get_queue (rdev->bdev); +struct backing_dev_info *bdi = &q->backing_dev_info;/* Note the '|| 1' - when read_balance prefers * non-congested targets, it can be removed */ if ((bits & (1 <<BDI_async_congested)) ||1 ) - ret |=bdi_congested (&q->backing_dev_info, bits); + ret |= group ?bdi_congested_group (bdi, bits, + page) :bdi_congested (bdi, bits);else - ret &=bdi_congested (&q->backing_dev_info, bits); + ret &= group ?bdi_congested_group (bdi, bits, + page) :bdi_congested (bdi, bits); } }rcu_read_unlock ();
drivers/md/raid10.c +4 -2
@@ -625,7 +625,7 @@ static void raid10_unplug(struct request_queue *q)md_wakeup_thread (mddev->thread); } -static int raid10_congested (void *data,int bits) +static int raid10_congested (void *data,int bits,struct page *page,int group) {mddev_t *mddev = data;conf_t *conf = mddev->private;
@@ -636,8 +636,10 @@ static int raid10_congested(void *data, int bits)mdk_rdev_t *rdev =rcu_dereference (conf->mirrors[i].rdev);if (rdev && !test_bit (Faulty, &rdev->flags)) {struct request_queue *q =bdev_get_queue (rdev->bdev); +struct backing_dev_info *bdi = &q->backing_dev_info; - ret |=bdi_congested (&q->backing_dev_info, bits); + ret |= group ?bdi_congested_group (bdi, bits, page) + :bdi_congested (bdi, bits); } }rcu_read_unlock ();
drivers/md/raid5.c +1 -1
@@ -3323,7 +3323,7 @@ static void raid5_unplug_device(struct request_queue *q)unplug_slaves (mddev); } -static int raid5_congested (void *data,int bits) +static int raid5_congested (void *data,int bits,struct page *page,int group) {mddev_t *mddev = data;raid5_conf_t *conf = mddev->private;
fs/afs/write.c +7 -1
@@ -455,7 +455,7 @@ int afs_writepage(struct page *page, struct writeback_control *wbc) } wbc->nr_to_write -= ret; -if (wbc->nonblocking &&bdi_write_congested (bdi)) +if (wbc->nonblocking &&bdi_or_group_write_congested (bdi, page)) wbc->encountered_congestion =1 ;_leave (" = 0" );
@@ -491,6 +491,12 @@ static int afs_writepages_region(struct address_space *mapping,return 0 ; } +if (wbc->nonblocking &&bdi_write_congested_group (bdi, page)) { + wbc->encountered_congestion =1 ; +page_cache_release (page); +break ; + } +/* at this point we hold neither mapping->tree_lock nor lock on * the page itself: the page may be truncated or invalidated * (changing page->mapping to NULL), or even swizzled back from
fs/btrfs/disk-io.c +4 -2
@@ -1249,7 +1249,8 @@ struct btrfs_root *btrfs_read_fs_root(struct btrfs_fs_info *fs_info,return root; } -static int btrfs_congested_fn (void *congested_data,int bdi_bits) +static int btrfs_congested_fn (void *congested_data,int bdi_bits, +struct page *page,int group) {struct btrfs_fs_info *info = (struct btrfs_fs_info *)congested_data;int ret =0 ;
@@ -1260,7 +1261,8 @@ static int btrfs_congested_fn(void *congested_data, int bdi_bits)if (!device->bdev)continue ; bdi =blk_get_backing_dev_info (device->bdev); -if (bdi &&bdi_congested (bdi, bdi_bits)) { +if (bdi && (group ?bdi_congested_group (bdi, bdi_bits, page) : +bdi_congested (bdi, bdi_bits))) { ret =1 ;break ; }
fs/btrfs/extent_io.c +12 -0
@@ -2368,6 +2368,18 @@ retry:unsigned i; scanned =1 ; + +/* +* If the io group page will go into is congested, bail out. +*/ +if (wbc->nonblocking + &&bdi_write_congested_group (bdi, pvec.pages[0 ])) { + wbc->encountered_congestion =1 ; + done =1 ; +pagevec_release (&pvec); +break ; + } +for (i =0 ; i < nr_pages; i++) {struct page *page = pvec.pages[i];
fs/btrfs/volumes.c +6 -2
@@ -165,6 +165,7 @@ static noinline int run_scheduled_bios(struct btrfs_device *device)unsigned long limit;unsigned long last_waited =0 ;int force_reg =0 ; +struct page *page; bdi =blk_get_backing_dev_info (device->bdev); fs_info = device->dev_root->fs_info;
@@ -276,8 +277,11 @@ loop_lock:* is now congested. Back off and let other work structs * run instead */ -if (pending &&bdi_write_congested (bdi) && batch_run >32 && - fs_info->fs_devices->open_devices >1 ) { +if (pending) + page =bio_iovec_idx (pending,0 )->bv_page; + +if (pending &&bdi_or_group_write_congested (bdi, page) && + num_run >32 && fs_info->fs_devices->open_devices >1 ) {struct io_context *ioc; ioc = current->io_context;
fs/cifs/file.c +11 -0
@@ -1470,6 +1470,17 @@ retry: n_iov =0 ; bytes_to_write =0 ; +/* +* If the io group page will go into is congested, bail out. +*/ +if (wbc->nonblocking && +bdi_write_congested_group (bdi, pvec.pages[0 ])) { + wbc->encountered_congestion =1 ; + done =1 ; +pagevec_release (&pvec); +break ; + } +for (i =0 ; i < nr_pages; i++) { page = pvec.pages[i];/*
fs/ext2/ialloc.c +1 -1
@@ -179,7 +179,7 @@ static void ext2_preread_inode(struct inode *inode)struct backing_dev_info *bdi; bdi = inode->i_mapping->backing_dev_info; -if (bdi_read_congested (bdi)) +if (bdi_or_group_read_congested (bdi,NULL ))return ;if (bdi_write_congested (bdi))return ;
fs/gfs2/aops.c +12 -0
@@ -371,6 +371,18 @@ retry: PAGECACHE_TAG_DIRTY, min(end - index, (pgoff_t)PAGEVEC_SIZE-1) + 1))) { scanned =1 ; + +/* +* If io group page belongs to is congested. bail out. +*/ +if (wbc->nonblocking + &&bdi_write_congested_group (bdi, pvec.pages[0 ])) { + wbc->encountered_congestion =1 ; + done =1 ; +pagevec_release (&pvec); +break ; + } + ret =gfs2_write_jdata_pagevec (mapping, wbc, &pvec, nr_pages, end);if (ret) done =1 ;
fs/nilfs2/segbuf.c +2 -1
@@ -266,8 +266,9 @@ static int nilfs_submit_seg_bio(struct nilfs_write_info *wi, int mode) {struct bio *bio = wi->bio;int err; +struct page *page =bio_iovec_idx (bio,0 )->bv_page; -if (wi->nbio >0 &&bdi_write_congested (wi->bdi)) { +if (wi->nbio >0 &&bdi_or_group_write_congested (wi->bdi, page)) {wait_for_completion (&wi->bio_event); wi->nbio--;if (unlikely (atomic_read (&wi->err))) {
fs/xfs/linux-2.6/xfs_aops.c +1 -1
@@ -891,7 +891,7 @@ xfs_convert_page( bdi = inode->i_mapping->backing_dev_info; wbc->nr_to_write--; -if (bdi_write_congested (bdi)) { +if (bdi_or_group_write_congested (bdi, page)) { wbc->encountered_congestion =1 ; done =1 ; }else if (wbc->nr_to_write <=0 ) {
fs/xfs/linux-2.6/xfs_buf.c +1 -1
@@ -714,7 +714,7 @@ xfs_buf_readahead(struct backing_dev_info *bdi; bdi = target->bt_mapping->backing_dev_info; -if (bdi_read_congested (bdi)) +if (bdi_or_group_read_congested (bdi,NULL ))return ; flags |= (XBF_TRYLOCK |XBF_ASYNC |XBF_READ_AHEAD );
include/linux/backing-dev.h +59 -2
79 lines not on this page · raw message
include/linux/blkdev.h +5 -0
@@ -886,6 +886,11 @@ static inline void blk_set_queue_congested(struct request_queue *q, int sync)set_bdi_congested (&q->backing_dev_info, sync); } +#ifdef CONFIG_GROUP_IOSCHED +extern int blk_queue_io_group_congested (struct backing_dev_info *bdi, +int bdi_bits,struct page *page); +#endif +extern void blk_start_queue (struct request_queue *q);extern void blk_stop_queue (struct request_queue *q);extern void blk_sync_queue (struct request_queue *q);
mm/backing-dev.c +62 -0
71 lines not on this page · raw message
mm/page-writeback.c +11 -0
@@ -985,6 +985,17 @@ retry:if (nr_pages ==0 )break ; +/* +* If the io group page will go into is congested, bail out. +*/ +if (wbc->nonblocking + &&bdi_write_congested_group (bdi, pvec.pages[0 ])) { + wbc->encountered_congestion =1 ; + done =1 ; +pagevec_release (&pvec); +break ; + } +for (i =0 ; i < nr_pages; i++) {struct page *page = pvec.pages[i];
mm/readahead.c +1 -1
@@ -542,7 +542,7 @@ page_cache_async_readahead(struct address_space *mapping,/* * Defer asynchronous read-ahead on IO congestion. */ -if (bdi_read_congested (mapping->backing_dev_info)) +if (bdi_or_group_read_congested (mapping->backing_dev_info,NULL ))return ;/* do read-ahead */
Signature · 1 line
1.6.0.6
Re: [PATCH 22/24] io-controller: Per io group bdi congestion interface
Vivek Goyal wrote:
...
Quoted 6 lines
(quoted diff) +6
-0+/* Set io group congestion on and off thresholds */ +void elv_io_group_congestion_threshold(struct request_queue *q, + struct io_group *iog) +{ + int nr; +
Review
Quoted 10 lines
(quoted diff) +10
-0+ nr = q->nr_group_requests - (q->nr_group_requests / 8) + 1; + if (nr > q->nr_group_requests) + nr = q->nr_group_requests; + iog->nr_congestion_on = nr; + + nr = q->nr_group_requests - (q->nr_group_requests / 8) + - (q->nr_group_requests / 16) - 1; + if (nr < 1) + nr = 1; + iog->nr_congestion_off = nr;
Vivek,
Since we determine whether io group is congenssed or not just by checking iog->nr_congestion_on, we can get rid of iog->nr_congestion_off here.
--- block/elevator-fq.c | 6 ------ block/elevator-fq.h | 3 +-- 2 files changed, 1 insertions(+), 8 deletions(-)
block/elevator-fq.c +0 -6
@@ -1267,12 +1267,6 @@ void elv_io_group_congestion_threshold(struct request_queue *q,if (nr > q->nr_group_requests) nr = q->nr_group_requests; iog->nr_congestion_on = nr; - - nr = q->nr_group_requests - (q->nr_group_requests /8 ) - - (q->nr_group_requests /16 ) -1 ; -if (nr <1 ) - nr =1 ; - iog->nr_congestion_off = nr; } static inline int elv_is_iog_congested(struct request_queue *q,
block/elevator-fq.h +1 -2
@@ -266,9 +266,8 @@ struct io_group {/* Single ioq per group, used for noop, deadline, anticipatory */ struct io_queue *ioq; -/* io group congestion on and off threshold for request descriptors */ +/* io group congestion on threshold for request descriptors */ unsigned int nr_congestion_on; -unsigned int nr_congestion_off;/* request list associated with the group */ struct request_list rl;
Signature · 1 line
1.5.4.rc3
A message that is not in the archive (sent off-list or not yet received): 1248467274-32073-23-git-send-email-vgoyal-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org
Re: [PATCH 22/24] io-controller: Per io group bdi congestion interface
Vivek Goyal wrote:
...
Quoted 6 lines
(quoted diff) +6
-0+/* Set io group congestion on and off thresholds */ +void elv_io_group_congestion_threshold(struct request_queue *q, + struct io_group *iog) +{ + int nr; +
Review
Quoted 10 lines
(quoted diff) +10
-0+ nr = q->nr_group_requests - (q->nr_group_requests / 8) + 1; + if (nr > q->nr_group_requests) + nr = q->nr_group_requests; + iog->nr_congestion_on = nr; + + nr = q->nr_group_requests - (q->nr_group_requests / 8) + - (q->nr_group_requests / 16) - 1; + if (nr < 1) + nr = 1; + iog->nr_congestion_off = nr;
Vivek,
Since we determine whether io group is congenssed or not just by checking iog->nr_congestion_on, we can get rid of iog->nr_congestion_off here.
--- block/elevator-fq.c | 6 ------ block/elevator-fq.h | 3 +-- 2 files changed, 1 insertions(+), 8 deletions(-)
block/elevator-fq.c +0 -6
@@ -1267,12 +1267,6 @@ void elv_io_group_congestion_threshold(struct request_queue *q,if (nr > q->nr_group_requests) nr = q->nr_group_requests; iog->nr_congestion_on = nr; - - nr = q->nr_group_requests - (q->nr_group_requests /8 ) - - (q->nr_group_requests /16 ) -1 ; -if (nr <1 ) - nr =1 ; - iog->nr_congestion_off = nr; } static inline int elv_is_iog_congested(struct request_queue *q,
block/elevator-fq.h +1 -2
@@ -266,9 +266,8 @@ struct io_group {/* Single ioq per group, used for noop, deadline, anticipatory */ struct io_queue *ioq; -/* io group congestion on and off threshold for request descriptors */ +/* io group congestion on threshold for request descriptors */ unsigned int nr_congestion_on; -unsigned int nr_congestion_off;/* request list associated with the group */ struct request_list rl;
Signature · 1 line
1.5.4.rc3
PATCH 23/24 io-controller: Support per cgroup per device weights and io class
In reply to Vivek GoyalThis patch enables per-cgroup per-device weight and ioprio_class handling. A new cgroup interface "policy" is introduced. You can make use of this file to configure weight and ioprio_class for each device in a given cgroup. The original "weight" and "ioprio_class" files are still available. If you don't do special configuration for a particular device, "weight" and "ioprio_class" are used as default values in this device.
You can use the following format to play with the new interface. #echo dev_major:dev_minor weight ioprio_class > /patch/to/cgroup/policy weight=0 means removing the policy for device.
Examples:
Configure weight=300 ioprio_class=2 on /dev/hdb (8:16) in this cgroup
# echo "8:16 300 2" > io.policy # cat io.policy dev weight class 8:16 300 2
Configure weight=500 ioprio_class=1 on /dev/hda (8:0) in this cgroup
# echo "8:0 500 1" > io.policy # cat io.policy dev weight class 8:0 500 1 8:16 300 2
Remove the policy for /dev/hda in this cgroup
# echo 8:0 0 1 > io.policy # cat io.policy dev weight class 8:16 300 2
Changelog (v1 -> v2)
- Rename some structures
- Use spin_lock_irqsave() and spin_lock_irqrestore() version to prevent from enabling the interrupts unconditionally.
- Fix policy setup bug when switching to another io scheduler.
- If a policy is available for a specific device, don't update weight and io class when writing "weight" and "iprio_class".
- Fix a bug when parsing policy string.
--- block/elevator-fq.c | 265 ++++++++++++++++++++++++++++++++++++++++++++++++++- block/elevator-fq.h | 10 ++ 2 files changed, 271 insertions(+), 4 deletions(-)
block/elevator-fq.c +261 -4
@@ -17,6 +17,7 @@#include <linux/blktrace_api.h> #include <linux/seq_file.h> #include <linux/biotrack.h> +#include <linux/genhd.h> /* Values taken from cfq */ const int elv_slice_sync =HZ /10 ;
@@ -1189,12 +1190,30 @@ bfq_init_entity(struct io_entity *entity, struct io_group *iog) entity->sched_data = &iog->sched_data; } -static void io_group_init_entity (struct io_cgroup *iocg,struct io_group *iog) +static struct io_policy_node *policy_search_node (const struct io_cgroup *iocg, +dev_t dev); + +static void +io_group_init_entity (struct io_cgroup *iocg,struct io_group *iog,dev_t dev) {struct io_entity *entity = &iog->entity; +struct io_policy_node *pn; +unsigned long flags; - entity->weight = iocg->weight; - entity->ioprio_class = entity->new_ioprio_class = iocg->ioprio_class; +spin_lock_irqsave (&iocg->lock, flags); + pn =policy_search_node (iocg, dev); +if (pn) { + entity->weight = pn->weight; + entity->new_weight = pn->weight; + entity->ioprio_class = pn->ioprio_class; + entity->new_ioprio_class = pn->ioprio_class; + }else { + entity->weight = iocg->weight; + entity->new_weight = iocg->weight; + entity->ioprio_class = iocg->ioprio_class; + entity->new_ioprio_class = iocg->ioprio_class; + } +spin_unlock_irqrestore (&iocg->lock, flags); entity->ioprio_changed =1 ; entity->my_sched_data = &iog->sched_data; }
@@ -1314,6 +1333,227 @@ io_cgroup_lookup_group(struct io_cgroup *iocg, void *key)return NULL ; } +static int io_cgroup_policy_read (struct cgroup *cgrp,struct cftype *cft, +struct seq_file *m) +{ +struct io_cgroup *iocg; +struct io_policy_node *pn; + + iocg =cgroup_to_io_cgroup (cgrp); + +if (list_empty (&iocg->policy_list)) + goto out; + +seq_printf (m,"dev\tweight\tclass\n" ); + +spin_lock_irq (&iocg->lock); +list_for_each_entry (pn, &iocg->policy_list, node) { +seq_printf (m,"%u:%u\t%u\t%hu\n" ,MAJOR (pn->dev), +MINOR (pn->dev), pn->weight, pn->ioprio_class); + } +spin_unlock_irq (&iocg->lock); +out: +return 0 ; +} + +static inline void policy_insert_node (struct io_cgroup *iocg, +struct io_policy_node *pn) +{ +list_add (&pn->node, &iocg->policy_list); +} + +/* Must be called with iocg->lock held */ +static inline void policy_delete_node (struct io_policy_node *pn) +{ +list_del (&pn->node); +} + +/* Must be called with iocg->lock held */ +static struct io_policy_node *policy_search_node (const struct io_cgroup *iocg, +dev_t dev) +{ +struct io_policy_node *pn; + +if (list_empty (&iocg->policy_list)) +return NULL ; + +list_for_each_entry (pn, &iocg->policy_list, node) { +if (pn->dev == dev) +return pn; + } + +return NULL ; +} + +static int check_dev_num (dev_t dev) +{ +int part =0 ; +struct gendisk *disk; + + disk =get_gendisk (dev, &part); +if (!disk || part) +return -ENODEV ; + +return 0 ; +} + +static int policy_parse_and_set (char *buf,struct io_policy_node *newpn) +{ +char *s[4 ], *p, *major_s =NULL , *minor_s =NULL ; +int ret; +unsigned long major, minor, temp; +int i =0 ; +dev_t dev; + +memset (s,0 ,sizeof (s)); +while ((p =strsep (&buf," " )) !=NULL ) { +if (!*p) +continue ; + s[i++] = p; + +/* Prevent from inputing too many things */ +if (i ==4 ) +break ; + } + +if (i !=3 ) +return -EINVAL ; + + p =strsep (&s[0 ],":" ); +if (p !=NULL ) + major_s = p; +else +return -EINVAL ; + + minor_s = s[0 ]; +if (!minor_s) +return -EINVAL ; + + ret =strict_strtoul (major_s,10 , &major); +if (ret) +return -EINVAL ; + + ret =strict_strtoul (minor_s,10 , &minor); +if (ret) +return -EINVAL ; + + dev =MKDEV (major, minor); + + ret =check_dev_num (dev); +if (ret) +return ret; + + newpn->dev = dev; + +if (s[1 ] ==NULL ) +return -EINVAL ; + + ret =strict_strtoul (s[1 ],10 , &temp); +if (ret || temp >WEIGHT_MAX ) +return -EINVAL ; + + newpn->weight = temp; + +if (s[2 ] ==NULL ) +return -EINVAL ; + + ret =strict_strtoul (s[2 ],10 , &temp); +if (ret || temp <IOPRIO_CLASS_RT || temp >IOPRIO_CLASS_IDLE ) +return -EINVAL ; + newpn->ioprio_class = temp; + +return 0 ; +} + +static int io_cgroup_policy_write (struct cgroup *cgrp,struct cftype *cft, +const char *buffer) +{ +struct io_cgroup *iocg; +struct io_policy_node *newpn, *pn; +char *buf; +int ret =0 ; +int keep_newpn =0 ; +struct hlist_node *n; +struct io_group *iog; + + buf =kstrdup (buffer,GFP_KERNEL ); +if (!buf) +return -ENOMEM ; + + newpn =kzalloc (sizeof (*newpn),GFP_KERNEL ); +if (!newpn) { + ret = -ENOMEM ; + goto free_buf; + } + + ret =policy_parse_and_set (buf, newpn); +if (ret) + goto free_newpn; + +if (!cgroup_lock_live_group (cgrp)) { + ret = -ENODEV ; + goto free_newpn; + } + + iocg =cgroup_to_io_cgroup (cgrp); +spin_lock_irq (&iocg->lock); + + pn =policy_search_node (iocg, newpn->dev); +if (!pn) { +if (newpn->weight !=0 ) { +policy_insert_node (iocg, newpn); + keep_newpn =1 ; + } + goto update_io_group; + } + +if (newpn->weight ==0 ) { +/* weight == 0 means deleteing a policy */ +policy_delete_node (pn); + goto update_io_group; + } + + pn->weight = newpn->weight; + pn->ioprio_class = newpn->ioprio_class; + +update_io_group: +hlist_for_each_entry (iog, n, &iocg->group_data, group_node) { +if (iog->dev == newpn->dev) { +if (newpn->weight) { + iog->entity.new_weight = newpn->weight; + iog->entity.new_ioprio_class = + newpn->ioprio_class; +/* +* iog weight and ioprio_class updating +* actually happens if ioprio_changed is set. +* So ensure ioprio_changed is not set until +* new weight and new ioprio_class are updated. +*/ +smp_wmb (); + iog->entity.ioprio_changed =1 ; + }else { + iog->entity.new_weight = iocg->weight; + iog->entity.new_ioprio_class = + iocg->ioprio_class; + +/* The same as above */ +smp_wmb (); + iog->entity.ioprio_changed =1 ; + } + } + } +spin_unlock_irq (&iocg->lock); + +cgroup_unlock (); + +free_newpn: +if (!keep_newpn) +kfree (newpn); +free_buf: +kfree (buf); +return ret; +} +#define SHOW_FUNCTION (__VAR) \ static u64 io_cgroup_##__VAR##_read(struct cgroup *cgroup, \ struct cftype *cftype) \
@@ -1346,6 +1586,7 @@ static int io_cgroup_##__VAR##_write(struct cgroup *cgroup, \struct io_cgroup *iocg; \struct io_group *iog; \struct hlist_node *n; \ +struct io_policy_node *pn; \ \if (val < (__MIN) || val > (__MAX)) \return -EINVAL ; \
@@ -1358,6 +1599,9 @@ static int io_cgroup_##__VAR##_write(struct cgroup *cgroup, \spin_lock_irq (&iocg->lock); \ iocg->__VAR = (unsigned long )val; \ hlist_for_each_entry(iog, n, &iocg->group_data, group_node) { \ + pn =policy_search_node (iocg, iog->dev); \ +if (pn) \ +continue ; \ iog->entity.new_##__VAR = (unsigned long)val; \smp_wmb (); \ iog->entity.ioprio_changed =1 ; \
@@ -1496,6 +1740,12 @@ static int io_cgroup_disk_dequeue_read(struct cgroup *cgroup,struct cftype bfqio_files[] = { { + .name ="policy" , + .read_seq_string = io_cgroup_policy_read, + .write_string = io_cgroup_policy_write, + .max_write_len =256 , + }, + { .name ="weight" , .read_u64 = io_cgroup_weight_read, .write_u64 = io_cgroup_weight_write,
@@ -1547,6 +1797,7 @@ static struct cgroup_subsys_state *iocg_create(struct cgroup_subsys *subsys,INIT_HLIST_HEAD (&iocg->group_data); iocg->weight =IO_DEFAULT_GRP_WEIGHT ; iocg->ioprio_class =IO_DEFAULT_GRP_CLASS ; +INIT_LIST_HEAD (&iocg->policy_list);return &iocg->css; }
@@ -1649,7 +1900,7 @@ io_group_chain_alloc(struct request_queue *q, void *key, struct cgroup *cgroup)sscanf (dev_name (bdi->dev),"%u:%u" , &major, &minor); iog->dev =MKDEV (major, minor); -io_group_init_entity (iocg, iog); +io_group_init_entity (iocg, iog, iog->dev); iog->my_entity = &iog->entity;atomic_set (&iog->ref,0 );
@@ -2121,6 +2372,7 @@ static void iocg_destroy(struct cgroup_subsys *subsys, struct cgroup *cgroup)struct io_group *iog;struct elv_fq_data *efqd;unsigned long uninitialized_var (flags ); +struct io_policy_node *pn, *pntmp;/* * io groups are linked in two lists. One list is maintained
@@ -2160,6 +2412,11 @@ remove_entry: goto remove_entry; done: +list_for_each_entry_safe (pn, pntmp, &iocg->policy_list, node) { +policy_delete_node (pn); +kfree (pn); + } +free_css_id (&io_subsys, &iocg->css);rcu_read_unlock ();BUG_ON (!hlist_empty (&iocg->group_data));
block/elevator-fq.h +10 -0
@@ -274,6 +274,13 @@ struct io_group {struct request_list rl; }; +struct io_policy_node { +struct list_head node; +dev_t dev; +unsigned int weight; +unsigned short ioprio_class; +}; +/** * struct io_cgroup - io cgroup data structure. * @css: subsystem state for io in the containing cgroup.
@@ -291,6 +298,9 @@ struct io_cgroup {unsigned int weight;unsigned short ioprio_class; +/* list of io_policy_node */ +struct list_head policy_list; +spinlock_t lock;struct hlist_head group_data; };
Signature · 1 line
1.6.0.6
PATCH 24/24 map sync requests to group using bio tracking info
In reply to Vivek Goyalo sync requests are mapped to the cgroup submitting tasks belongs to. This is an experimental patch where sync requets can be mapped to the group using bio tracking information and not submitting task's context.
o This patch implements this functionality only for noop, deadline and AS.
o Introducing this patch for experimental purposes to see if it is really meaningful or not.
o This behavior is turned on by setting /sys/block/<disk>/queue/iosched/map_sync varibale.
--- block/as-iosched.c | 1 + block/deadline-iosched.c | 1 + block/elevator-fq.c | 7 ++++++- block/elevator-fq.h | 9 +++++++++ block/noop-iosched.c | 1 + 5 files changed, 18 insertions(+), 1 deletions(-)
block/as-iosched.c +1 -0
@@ -1768,6 +1768,7 @@ static struct elv_fs_entry as_attrs[] = {ELV_ATTR (fairness),ELV_ATTR (slice_sync),ELV_ATTR (group_idle), +ELV_ATTR (map_sync), #endif __ATTR_NULL };
block/deadline-iosched.c +1 -0
@@ -465,6 +465,7 @@ static struct elv_fs_entry deadline_attrs[] = {ELV_ATTR (fairness),ELV_ATTR (slice_sync),ELV_ATTR (group_idle), +ELV_ATTR (map_sync), #endif __ATTR_NULL };
block/elevator-fq.c +6 -1
@@ -2073,6 +2073,7 @@ struct io_group *io_get_io_group_bio(struct request_queue *q, struct bio *bio, int create) {struct page *page =NULL ; +struct elv_fq_data *efqd = &q->elevator->efqd;/* * Determine the group from task context. Even calls from
@@ -2091,7 +2092,7 @@ struct io_group *io_get_io_group_bio(struct request_queue *q, struct bio *bio, }/* Map the sync bio to the right group using task context */ -if (elv_bio_sync (bio)) +if (elv_bio_sync (bio) && !efqd->map_sync) goto sync;#ifdef CONFIG_TRACK_ASYNC_CONTEXT
@@ -2812,6 +2813,8 @@ SHOW_FUNCTION(elv_slice_async_show, efqd->elv_slice[0], 1);EXPORT_SYMBOL (elv_slice_async_show);SHOW_FUNCTION (elv_fairness_show, efqd->fairness,0 );EXPORT_SYMBOL (elv_fairness_show); +SHOW_FUNCTION (elv_map_sync_show, efqd->map_sync,0 ); +EXPORT_SYMBOL (elv_map_sync_show);#undef SHOW_FUNCTION#define STORE_FUNCTION (__FUNC, __PTR,MIN ,MAX , __CONV) \
@@ -2838,6 +2841,8 @@ STORE_FUNCTION(elv_slice_async_store, &efqd->elv_slice[0], 1, UINT_MAX, 1);EXPORT_SYMBOL (elv_slice_async_store);STORE_FUNCTION (elv_fairness_store, &efqd->fairness,0 ,1 ,0 );EXPORT_SYMBOL (elv_fairness_store); +STORE_FUNCTION (elv_map_sync_store, &efqd->map_sync,0 ,1 ,0 ); +EXPORT_SYMBOL (elv_map_sync_store);#undef STORE_FUNCTIONvoid elv_schedule_dispatch (struct request_queue *q)
block/elevator-fq.h +9 -0
@@ -347,6 +347,12 @@ struct elv_fq_data {* queue before new queue is scheduled in */ unsigned int fairness; + +/* +* Get io group bio belongs to from bio and not from submitting task +* context +*/ +unsigned int map_sync; };/* Logging facilities. */
@@ -551,6 +557,9 @@ extern void elv_put_iog(struct io_group *iog);extern ssize_t elv_group_idle_show (struct elevator_queue *q,char *name);extern ssize_t elv_group_idle_store (struct elevator_queue *q,const char *name,size_t count); +extern ssize_t elv_map_sync_show (struct elevator_queue *q,char *name); +extern ssize_t elv_map_sync_store (struct elevator_queue *q,const char *name, +size_t count);static inline void elv_get_iog (struct io_group *iog) {atomic_inc (&iog->ref);
block/noop-iosched.c +1 -0
@@ -87,6 +87,7 @@ static struct elv_fs_entry noop_attrs[] = {ELV_ATTR (fairness),ELV_ATTR (slice_sync),ELV_ATTR (group_idle), +ELV_ATTR (map_sync), __ATTR_NULL };#endif
Signature · 1 line
1.6.0.6
Hi Vivek,
Here are some test results for normal reads and write for IO Controller V7 by fio. Tested with "fairness == 0". It seems performance gets better comparing with V6.
Mode Normal read | Random read | Normal write | Random write | Direct read | Direct Write
2.6.31-rc1 71,613KiB/s 3,606KiB/s 66,250KiB/s 9,420KiB/s 51,535KiB/s 55,752KiB/s
V7 70,540KiB/s 3,551KiB/s 64,548KiB/s 9,677KiB/s 53,530KiB/s 54,145KiB/s
Performance -1.5% -1.5% -2.6% +2.7% +3.9% -2.9%
Quoted 402 lines · Vivek Goyal
Hi All,
Here is the V7 of the IO controller patches generated on top of 2.6.31-rc4.
For ease of patching, a consolidated patch is available here.
http://people.redhat.com/~vgoyal/io-controller/io-scheduler-based-io-controller-v7.patch
Previous versions of the patches was posted here.
(V1) http://lkml.org/lkml/2009/3/11/486
(V2) http://lkml.org/lkml/2009/5/5/275
(V3) http://lkml.org/lkml/2009/5/26/472
(V4) http://lkml.org/lkml/2009/6/8/580
(V5) http://lkml.org/lkml/2009/6/19/279
(V6) http://lkml.org/lkml/2009/7/2/369Changes from V6
===============
- Introduced the notion ofgroup_idlingwhere we idle for next request to come from the same group before we expire it. It is along the lines of cfq'sslice_idlething to provide fairness. Switching to group idling now helps in the sense that we don't have to rely whether queue idling was turned on or not by CFQ. It becomes too much of debugging pain with different work loads and different kind of storage media. Introduction ofgroup_idleshould help.- Moved some of the code like dynamic queue idling update, arming queue idling timer, keeping track of average think time etc back to CFQ. With group idling we don't need it now. Reduce the amount of change.
- Enabled cfq's close cooperator functionality in groups. So far this worked only in root group. Now it should work in non-root groups also.
- Got rid of the patch where we calculated disk time based on average disk rate in some circumstances. It was giving bad numbers in early queue deletion cases. Also did not think that it was helping a lot. Remvoed it for the time being.
- Added an experimental patch to map sync requests using bio tracking info and not task context. This is only for noop, deadline and AS.
- Got rid of experimental patch of idling for async queues. Don't think it was helping.
- Got rid of
wait_busyandwait_busy_donelogic from queue. Instead implemented it for groups.- Introduced
oom_ioqto accomodateoom_cfqqchange recently.- Broke-up
elv_init_ioq()fn into smaller functions. It had 7 arguments and looked complicated.- Fixed a bug in
blk_queue_io_group_congested(). Thanks to Munehiro Ikeda.- Merged gui's patch to fix the cgroup file format issue.
- Merged gui's patch to update per group congestion limit when
q->nr_group_requestsis updated.- Fixed a bug where close cooperation will not work if we wait for all the requests to finish from previous queue.
- Fixed group deletion accouting where deletion from idle tree were also appearing in the log.
- Got rid of
busy_rt_queuesinfrastructure.- Got rid of
elv_ioq_request_dispatched(). An helper function just to increment a variable.Limitations
===========- This IO controller provides the bandwidth control at the IO scheduler level (leaf node in stacked hiearchy of logical devices). So there can be cases (depending on configuration) where application does not see proportional BW division at higher logical level device.
LWN has written an article about the issue here.
http://lwn.net/Articles/332839/How to solve the issue of fairness at higher level logical devices
==================================================================
(Do we really need it? That's not where the contention for resources is.)Couple of suggestions have come forward.
- Implement IO control at IO scheduler layer and then with the help of some daemon, adjust the weight on underlying devices dynamiclly, depending on what kind of BW gurantees are to be achieved at higher level logical block devices.
- Also implement a higher level IO controller along with IO scheduler based controller and let user choose one depending on his needs.
A higher level controller does not know about the assumptions/policies of unerldying IO scheduler, hence it has the potential to break down the IO scheduler's policy with-in cgroup. A lower level controller can work with IO scheduler much more closely and efficiently.
Other active IO controller developments
=======================================IO throttling
-------------This is a max bandwidth controller and not the proportional one. Secondly it is a second level controller which can break the IO scheduler's policy/assumtions with-in cgroup.
dm-ioband
---------This is a proportional bandwidth controller implemented as device mapper driver. It is also a second level controller which can break the IO scheduler's policy/assumptions with-in cgroup.
TODO
====
- code cleanups, testing, bug fixing, optimizations, benchmarking etc...Testing
=======I have been able to do some testing as follows. All my testing is with ext3 file system with a SATA drive which supports queue depth of 31.
Test1 (Isolation between two KVM virtual machines)
==================================================
Created two KVM virtual machines. Partitioned a disk on host in two partitions and gave one partition to each virtual machine. Put both the virtual machines in two different cgroup of weight 1000 and 500 each. Virtual machines created ext3 file system on the partitions exported from host and did buffered writes. Host seems writes as synchronous and virtual machine with higher weight gets double the disk time of virtual machine of lower weight. Used deadline scheduler in this test case.Some more details about configuration are in documentation patch.
Test2 (Fairness for synchronous reads)
======================================
- Two dd in two cgroups with cgrop weights 1000 and 500. Ran two "dd" in those cgroups (With CFQ scheduler and /sys/block/<device>/queue/fairness = 1)Higher weight dd finishes first and at that point of time my script takes care of reading cgroup files io.disk_time and io.disk_sectors for both the groups and display the results.
dd if=/mnt/$BLOCKDEV/zerofile1 of=/dev/null &
dd if=/mnt/$BLOCKDEV/zerofile2 of=/dev/null &234179072 bytes (234 MB) copied, 3.9065 s, 59.9 MB/s 234179072 bytes (234 MB) copied, 5.19232 s, 45.1 MB/s
group1 time=8 16 2471 group1 sectors=8 16 457840 group2 time=8 16 1220 group2 sectors=8 16 225736
First two fields in time and sectors statistics represent major and minor number of the device. Third field represents disk time in milliseconds and number of sectors transferred respectively.
This patchset tries to provide fairness in terms of disk time received. group1 got almost double of group2 disk time (At the time of first dd finish). These time and sectors statistics can be read using io.disk_time and io.disk_sector files in cgroup. More about it in documentation file.
Test3 (Reader Vs Buffered Writes)
================================
Buffered writes can be problematic and can overwhelm readers, especially with noop and deadline. IO controller can provide isolation between readers and buffered (async) writers.First I ran the test without io controller to see the severity of the issue. Ran a hostile writer and then after 10 seconds started a reader and then monitored the completion time of reader. Reader reads a 256 MB file. Tested this with noop scheduler.
sample script
------------
sync
echo 3 >/proc/sys/vm/drop_caches
time dd if=/dev/zero of=/mnt/sdb/reader-writer-zerofile bs=4K count=2097152 conv=fdatasync &
sleep 10
time dd if=/mnt/sdb/256M-file of=/dev/null &Results
-------
8589934592 bytes (8.6 GB) copied, 106.045 s, 81.0 MB/s (Writer) 268435456 bytes (268 MB) copied, 96.5237 s, 2.8 MB/s (Reader)Now it was time to test io controller whether it can provide isolation between readers and writers with noop. I created two cgroups of weight 1000 each and put reader in group1 and writer in group 2 and ran the test again. Upon comletion of reader, my scripts read io.dis_time and io.disk_group cgroup files to get an estimate how much disk time each group got and how many sectors each group did IO for.
For more accurate accounting of disk time for buffered writes with queuing hardware I had to set /sys/block/<disk>/queue/iosched/fairness to "1".
sample script
-------------
echo $$ > /cgroup/bfqio/test2/tasks
dd if=/dev/zero of=/mnt/$BLOCKDEV/testzerofile bs=4K count=2097152 & sleep 10
echo noop >/sys/block/$BLOCKDEV/queue/scheduler
echo 1 >/sys/block/$BLOCKDEV/queue/iosched/fairnessecho $$ > /cgroup/bfqio/test1/tasks
dd if=/mnt/$BLOCKDEV/256M-file of=/dev/null &
wait $!
# Some code for reading cgroup files upon completion of reader.
-------------------------Results
=======
268435456 bytes (268 MB) copied, 6.65819 s, 40.3 MB/s (Reader)group1 time=8 16 3063 group1 sectors=8 16 524808
group2 time=8 16 3071 group2 sectors=8 16 441752Note, reader finishes now much lesser time and both group1 and group2 got almost 3 seconds of disk time. Hence io-controller provides isolation from buffered writes.
Test4 (AIO)
===========AIO reads
-----------
Set up two fio, AIO read jobs in two cgroup with weight 1000 and 500 respectively. I am using cfq scheduler. Following are some lines from my test script.---------------------------------------------------------------
echo 1000 > /cgroup/bfqio/test1/io.weight
echo 500 > /cgroup/bfqio/test2/io.weightfio_args="--ioengine=libaio --rw=read --size=512M --direct=1" echo 1 >
/sys/block/$BLOCKDEV/queue/iosched/fairnessecho $$ > /cgroup/bfqio/test1/tasks
fio $fio_args --name=test1 --directory=/mnt/$BLOCKDEV/fio1/ --output=/mnt/$BLOCKDEV/fio1/test1.log --exec_postrun="../read-and-display-group-stats.sh $maj_dev $minor_dev" &echo $$ > /cgroup/bfqio/test2/tasks
fio $fio_args --name=test2 --directory=/mnt/$BLOCKDEV/fio2/ --output=/mnt/$BLOCKDEV/fio2/test2.log &
----------------------------------------------------------------test1 and test2 are two groups with weight 1000 and 500 respectively. "
read-and-display-group-stats.sh" is one small script which reads the test1 and test2 cgroup files to determine how much disk time each group got till first fio job finished.Results
------
test1 statistics: time=8 16 22403 sectors=8 16 1049640
test2 statistics: time=8 16 11400 sectors=8 16 552864Above shows that by the time first fio (higher weight), finished, group test1 got 22403 ms of disk time and group test2 got 11400 ms of disk time. similarly the statistics for number of sectors transferred are also shown.
Note that disk time given to group test1 is almost double of group2 disk time.
AIO writes
----------
Set up two fio, AIO direct write jobs in two cgroup with weight 1000 and 500 respectively. I am using cfq scheduler. Following are some lines from my test script.------------------------------------------------
echo 1000 > /cgroup/bfqio/test1/io.weight
echo 500 > /cgroup/bfqio/test2/io.weight fio_args="--ioengine=libaio --rw=write --size=512M --direct=1"echo 1 >
/sys/block/$BLOCKDEV/queue/iosched/fairnessecho $$ > /cgroup/bfqio/test1/tasks
fio $fio_args --name=test1 --directory=/mnt/$BLOCKDEV/fio1/ --output=/mnt/$BLOCKDEV/fio1/test1.log --exec_postrun="../read-and-display-group-stats.sh $maj_dev $minor_dev" &echo $$ > /cgroup/bfqio/test2/tasks
fio $fio_args --name=test2 --directory=/mnt/$BLOCKDEV/fio2/ --output=/mnt/$BLOCKDEV/fio2/test2.log &
-------------------------------------------------test1 and test2 are two groups with weight 1000 and 500 respectively. "
read-and-display-group-stats.sh" is one small script which reads the test1 and test2 cgroup files to determine how much disk time each group got till first fio job finished.Following are the results.
test1 statistics: time=8 16 29085 sectors=8 16 1049656
test2 statistics: time=8 16 14652 sectors=8 16 516728Above shows that by the time first fio (higher weight), finished, group test1 got 28085 ms of disk time and group test2 got 14652 ms of disk time. similarly the statistics for number of sectors transferred are also shown.
Note that disk time given to group test1 is almost double of group2 disk time.
Test5 (Fairness for async writes, Buffered Write Vs Buffered Write)
===================================================================
Fairness for async writes is tricky and biggest reason is that async writes are cached in higher layers (page cahe) as well as possibly in file system layer also (btrfs, xfs etc), and are dispatched to lower layers not necessarily in proportional manner.For example, consider two dd threads reading
/dev/zeroas input file and doing writes of huge files. Very soon we will crossvm_dirty_ratioand dd thread will be forced to write out some pages to disk before more pages can be dirtied. But not necessarily dirty pages of same thread are picked. It can very well pick the inode of lesser priority dd thread and do some writeout. So effectively higher weight dd is doing writeouts of lower weight dd pages and we don't see service differentation.IOW, the core problem with async write fairness is that higher weight thread does not throw enought IO traffic at IO controller to keep the queue continuously backlogged. In my testing, there are many .2 to .8 second intervals where higher weight queue is empty and in that duration lower weight queue get lots of job done giving the impression that there was no service differentiation.
In summary, from IO controller point of view async writes support is there. Because page cache has not been designed in such a manner that higher prio/weight writer can do more write out as compared to lower prio/weight writer, gettting service differentiation is hard and it is visible in some cases and not visible in some cases.
Do we really care that much for fairness among two writer cgroups? One can choose to do direct writes or sync writes if fairness for writes really matters for him.
Following is the only case where it is hard to ensure fairness between cgroups.
- Buffered writes Vs Buffered Writes.
So to test async writes I created two partitions on a disk and created ext3 file systems on both the partitions. Also created two cgroups and generated lots of write traffic in two cgroups (50 fio threads) and watched the disk time statistics in respective cgroups at the interval of 2 seconds. Thanks to ryo tsuruta for the test case.
*****************************************************************
sync
echo 3 >/proc/sys/vm/drop_cachesfio_args="--size=64m --rw=write --numjobs=50 --group_reporting"
echo $$ > /cgroup/bfqio/test1/tasks
fio $fio_args --name=test1 --directory=/mnt/sdd1/fio/ --output=/mnt/sdd1/fio/test1.log &echo $$ > /cgroup/bfqio/test2/tasks
fio $fio_args --name=test2 --directory=/mnt/sdd2/fio/ --output=/mnt/sdd2/fio/test2.log &
***********************************************************************And watched the disk time and sector statistics for the both the cgroups every 2 seconds using a script. How is snippet from output.
test1 statistics: time=8 48 1315 sectors=8 48 55776 dq=8 48 1
test2 statistics: time=8 48 633 sectors=8 48 14720 dq=8 48 2test1 statistics: time=8 48 5586 sectors=8 48 339064 dq=8 48 2
test2 statistics: time=8 48 2985 sectors=8 48 146656 dq=8 48 3test1 statistics: time=8 48 9935 sectors=8 48 628728 dq=8 48 3
test2 statistics: time=8 48 5265 sectors=8 48 278688 dq=8 48 4test1 statistics: time=8 48 14156 sectors=8 48 932488 dq=8 48 6
test2 statistics: time=8 48 7646 sectors=8 48 412704 dq=8 48 7test1 statistics: time=8 48 18141 sectors=8 48 1231488 dq=8 48 10
test2 statistics: time=8 48 9820 sectors=8 48 548400 dq=8 48 8test1 statistics: time=8 48 21953 sectors=8 48 1485632 dq=8 48 13
test2 statistics: time=8 48 12394 sectors=8 48 698288 dq=8 48 10test1 statistics: time=8 48 25167 sectors=8 48 1705264 dq=8 48 13
test2 statistics: time=8 48 14042 sectors=8 48 817808 dq=8 48 10First two fields in time and sectors statistics represent major and minor number of the device. Third field represents disk time in milliseconds and number of sectors transferred respectively.
So disk time consumed by group1 is almost double of group2 in this case.
Your feedback is welcome.
Thanks
Vivek
Signature · 2 lines
Regards Gui Jianfeng
Quoted 5 lines · Gui Jianfeng
Hi Vivek,
Here are some test results for normal reads and write for IO Controller V7 by fio. Tested with "fairness == 0". It seems performance gets better comparing with V6.
Quoted 8 lines
Mode Normal read | Random read | Normal write | Random write | Direct read | Direct Write
2.6.31-rc1 71,613KiB/s 3,606KiB/s 66,250KiB/s 9,420KiB/s 51,535KiB/s 55,752KiB/s
V7 70,540KiB/s 3,551KiB/s 64,548KiB/s 9,677KiB/s 53,530KiB/s 54,145KiB/s
Performance -1.5% -1.5% -2.6% +2.7% +3.9% -2.9%
Thanks Gui. Can you also try V7 with CONFIG_TRACK_ASYNC_CONTEXT=n. I tried that and I got better results for buffered writes.
In my testing I still see some performance regression for buffered writes which goes away if I disable group io scheduling and just use flat mode.
I will spend more time to find out where it is coming from.
Thanks
Vivek
Quoted 408 lines
Quoted 402 lines · Vivek Goyal
Hi All,
Here is the V7 of the IO controller patches generated on top of 2.6.31-rc4.
For ease of patching, a consolidated patch is available here.
http://people.redhat.com/~vgoyal/io-controller/io-scheduler-based-io-controller-v7.patch
Previous versions of the patches was posted here.
(V1) http://lkml.org/lkml/2009/3/11/486
(V2) http://lkml.org/lkml/2009/5/5/275
(V3) http://lkml.org/lkml/2009/5/26/472
(V4) http://lkml.org/lkml/2009/6/8/580
(V5) http://lkml.org/lkml/2009/6/19/279
(V6) http://lkml.org/lkml/2009/7/2/369Changes from V6
===============
- Introduced the notion ofgroup_idlingwhere we idle for next request to come from the same group before we expire it. It is along the lines of cfq'sslice_idlething to provide fairness. Switching to group idling now helps in the sense that we don't have to rely whether queue idling was turned on or not by CFQ. It becomes too much of debugging pain with different work loads and different kind of storage media. Introduction ofgroup_idleshould help.- Moved some of the code like dynamic queue idling update, arming queue idling timer, keeping track of average think time etc back to CFQ. With group idling we don't need it now. Reduce the amount of change.
- Enabled cfq's close cooperator functionality in groups. So far this worked only in root group. Now it should work in non-root groups also.
- Got rid of the patch where we calculated disk time based on average disk rate in some circumstances. It was giving bad numbers in early queue deletion cases. Also did not think that it was helping a lot. Remvoed it for the time being.
- Added an experimental patch to map sync requests using bio tracking info and not task context. This is only for noop, deadline and AS.
- Got rid of experimental patch of idling for async queues. Don't think it was helping.
- Got rid of
wait_busyandwait_busy_donelogic from queue. Instead implemented it for groups.- Introduced
oom_ioqto accomodateoom_cfqqchange recently.- Broke-up
elv_init_ioq()fn into smaller functions. It had 7 arguments and looked complicated.- Fixed a bug in
blk_queue_io_group_congested(). Thanks to Munehiro Ikeda.- Merged gui's patch to fix the cgroup file format issue.
- Merged gui's patch to update per group congestion limit when
q->nr_group_requestsis updated.- Fixed a bug where close cooperation will not work if we wait for all the requests to finish from previous queue.
- Fixed group deletion accouting where deletion from idle tree were also appearing in the log.
- Got rid of
busy_rt_queuesinfrastructure.- Got rid of
elv_ioq_request_dispatched(). An helper function just to increment a variable.Limitations
===========- This IO controller provides the bandwidth control at the IO scheduler level (leaf node in stacked hiearchy of logical devices). So there can be cases (depending on configuration) where application does not see proportional BW division at higher logical level device.
LWN has written an article about the issue here.
http://lwn.net/Articles/332839/How to solve the issue of fairness at higher level logical devices
==================================================================
(Do we really need it? That's not where the contention for resources is.)Couple of suggestions have come forward.
- Implement IO control at IO scheduler layer and then with the help of some daemon, adjust the weight on underlying devices dynamiclly, depending on what kind of BW gurantees are to be achieved at higher level logical block devices.
- Also implement a higher level IO controller along with IO scheduler based controller and let user choose one depending on his needs.
A higher level controller does not know about the assumptions/policies of unerldying IO scheduler, hence it has the potential to break down the IO scheduler's policy with-in cgroup. A lower level controller can work with IO scheduler much more closely and efficiently.
Other active IO controller developments
=======================================IO throttling
-------------This is a max bandwidth controller and not the proportional one. Secondly it is a second level controller which can break the IO scheduler's policy/assumtions with-in cgroup.
dm-ioband
---------This is a proportional bandwidth controller implemented as device mapper driver. It is also a second level controller which can break the IO scheduler's policy/assumptions with-in cgroup.
TODO
====
- code cleanups, testing, bug fixing, optimizations, benchmarking etc...Testing
=======I have been able to do some testing as follows. All my testing is with ext3 file system with a SATA drive which supports queue depth of 31.
Test1 (Isolation between two KVM virtual machines)
==================================================
Created two KVM virtual machines. Partitioned a disk on host in two partitions and gave one partition to each virtual machine. Put both the virtual machines in two different cgroup of weight 1000 and 500 each. Virtual machines created ext3 file system on the partitions exported from host and did buffered writes. Host seems writes as synchronous and virtual machine with higher weight gets double the disk time of virtual machine of lower weight. Used deadline scheduler in this test case.Some more details about configuration are in documentation patch.
Test2 (Fairness for synchronous reads)
======================================
- Two dd in two cgroups with cgrop weights 1000 and 500. Ran two "dd" in those cgroups (With CFQ scheduler and /sys/block/<device>/queue/fairness = 1)Higher weight dd finishes first and at that point of time my script takes care of reading cgroup files io.disk_time and io.disk_sectors for both the groups and display the results.
dd if=/mnt/$BLOCKDEV/zerofile1 of=/dev/null &
dd if=/mnt/$BLOCKDEV/zerofile2 of=/dev/null &234179072 bytes (234 MB) copied, 3.9065 s, 59.9 MB/s 234179072 bytes (234 MB) copied, 5.19232 s, 45.1 MB/s
group1 time=8 16 2471 group1 sectors=8 16 457840 group2 time=8 16 1220 group2 sectors=8 16 225736
First two fields in time and sectors statistics represent major and minor number of the device. Third field represents disk time in milliseconds and number of sectors transferred respectively.
This patchset tries to provide fairness in terms of disk time received. group1 got almost double of group2 disk time (At the time of first dd finish). These time and sectors statistics can be read using io.disk_time and io.disk_sector files in cgroup. More about it in documentation file.
Test3 (Reader Vs Buffered Writes)
================================
Buffered writes can be problematic and can overwhelm readers, especially with noop and deadline. IO controller can provide isolation between readers and buffered (async) writers.First I ran the test without io controller to see the severity of the issue. Ran a hostile writer and then after 10 seconds started a reader and then monitored the completion time of reader. Reader reads a 256 MB file. Tested this with noop scheduler.
sample script
------------
sync
echo 3 >/proc/sys/vm/drop_caches
time dd if=/dev/zero of=/mnt/sdb/reader-writer-zerofile bs=4K count=2097152 conv=fdatasync &
sleep 10
time dd if=/mnt/sdb/256M-file of=/dev/null &Results
-------
8589934592 bytes (8.6 GB) copied, 106.045 s, 81.0 MB/s (Writer) 268435456 bytes (268 MB) copied, 96.5237 s, 2.8 MB/s (Reader)Now it was time to test io controller whether it can provide isolation between readers and writers with noop. I created two cgroups of weight 1000 each and put reader in group1 and writer in group 2 and ran the test again. Upon comletion of reader, my scripts read io.dis_time and io.disk_group cgroup files to get an estimate how much disk time each group got and how many sectors each group did IO for.
For more accurate accounting of disk time for buffered writes with queuing hardware I had to set /sys/block/<disk>/queue/iosched/fairness to "1".
sample script
-------------
echo $$ > /cgroup/bfqio/test2/tasks
dd if=/dev/zero of=/mnt/$BLOCKDEV/testzerofile bs=4K count=2097152 & sleep 10
echo noop >/sys/block/$BLOCKDEV/queue/scheduler
echo 1 >/sys/block/$BLOCKDEV/queue/iosched/fairnessecho $$ > /cgroup/bfqio/test1/tasks
dd if=/mnt/$BLOCKDEV/256M-file of=/dev/null &
wait $!
# Some code for reading cgroup files upon completion of reader.
-------------------------Results
=======
268435456 bytes (268 MB) copied, 6.65819 s, 40.3 MB/s (Reader)group1 time=8 16 3063 group1 sectors=8 16 524808
group2 time=8 16 3071 group2 sectors=8 16 441752Note, reader finishes now much lesser time and both group1 and group2 got almost 3 seconds of disk time. Hence io-controller provides isolation from buffered writes.
Test4 (AIO)
===========AIO reads
-----------
Set up two fio, AIO read jobs in two cgroup with weight 1000 and 500 respectively. I am using cfq scheduler. Following are some lines from my test script.---------------------------------------------------------------
echo 1000 > /cgroup/bfqio/test1/io.weight
echo 500 > /cgroup/bfqio/test2/io.weightfio_args="--ioengine=libaio --rw=read --size=512M --direct=1" echo 1 >
/sys/block/$BLOCKDEV/queue/iosched/fairnessecho $$ > /cgroup/bfqio/test1/tasks
fio $fio_args --name=test1 --directory=/mnt/$BLOCKDEV/fio1/ --output=/mnt/$BLOCKDEV/fio1/test1.log --exec_postrun="../read-and-display-group-stats.sh $maj_dev $minor_dev" &echo $$ > /cgroup/bfqio/test2/tasks
fio $fio_args --name=test2 --directory=/mnt/$BLOCKDEV/fio2/ --output=/mnt/$BLOCKDEV/fio2/test2.log &
----------------------------------------------------------------test1 and test2 are two groups with weight 1000 and 500 respectively. "
read-and-display-group-stats.sh" is one small script which reads the test1 and test2 cgroup files to determine how much disk time each group got till first fio job finished.Results
------
test1 statistics: time=8 16 22403 sectors=8 16 1049640
test2 statistics: time=8 16 11400 sectors=8 16 552864Above shows that by the time first fio (higher weight), finished, group test1 got 22403 ms of disk time and group test2 got 11400 ms of disk time. similarly the statistics for number of sectors transferred are also shown.
Note that disk time given to group test1 is almost double of group2 disk time.
AIO writes
----------
Set up two fio, AIO direct write jobs in two cgroup with weight 1000 and 500 respectively. I am using cfq scheduler. Following are some lines from my test script.------------------------------------------------
echo 1000 > /cgroup/bfqio/test1/io.weight
echo 500 > /cgroup/bfqio/test2/io.weight fio_args="--ioengine=libaio --rw=write --size=512M --direct=1"echo 1 >
/sys/block/$BLOCKDEV/queue/iosched/fairnessecho $$ > /cgroup/bfqio/test1/tasks
fio $fio_args --name=test1 --directory=/mnt/$BLOCKDEV/fio1/ --output=/mnt/$BLOCKDEV/fio1/test1.log --exec_postrun="../read-and-display-group-stats.sh $maj_dev $minor_dev" &echo $$ > /cgroup/bfqio/test2/tasks
fio $fio_args --name=test2 --directory=/mnt/$BLOCKDEV/fio2/ --output=/mnt/$BLOCKDEV/fio2/test2.log &
-------------------------------------------------test1 and test2 are two groups with weight 1000 and 500 respectively. "
read-and-display-group-stats.sh" is one small script which reads the test1 and test2 cgroup files to determine how much disk time each group got till first fio job finished.Following are the results.
test1 statistics: time=8 16 29085 sectors=8 16 1049656
test2 statistics: time=8 16 14652 sectors=8 16 516728Above shows that by the time first fio (higher weight), finished, group test1 got 28085 ms of disk time and group test2 got 14652 ms of disk time. similarly the statistics for number of sectors transferred are also shown.
Note that disk time given to group test1 is almost double of group2 disk time.
Test5 (Fairness for async writes, Buffered Write Vs Buffered Write)
===================================================================
Fairness for async writes is tricky and biggest reason is that async writes are cached in higher layers (page cahe) as well as possibly in file system layer also (btrfs, xfs etc), and are dispatched to lower layers not necessarily in proportional manner.For example, consider two dd threads reading
/dev/zeroas input file and doing writes of huge files. Very soon we will crossvm_dirty_ratioand dd thread will be forced to write out some pages to disk before more pages can be dirtied. But not necessarily dirty pages of same thread are picked. It can very well pick the inode of lesser priority dd thread and do some writeout. So effectively higher weight dd is doing writeouts of lower weight dd pages and we don't see service differentation.IOW, the core problem with async write fairness is that higher weight thread does not throw enought IO traffic at IO controller to keep the queue continuously backlogged. In my testing, there are many .2 to .8 second intervals where higher weight queue is empty and in that duration lower weight queue get lots of job done giving the impression that there was no service differentiation.
In summary, from IO controller point of view async writes support is there. Because page cache has not been designed in such a manner that higher prio/weight writer can do more write out as compared to lower prio/weight writer, gettting service differentiation is hard and it is visible in some cases and not visible in some cases.
Do we really care that much for fairness among two writer cgroups? One can choose to do direct writes or sync writes if fairness for writes really matters for him.
Following is the only case where it is hard to ensure fairness between cgroups.
- Buffered writes Vs Buffered Writes.
So to test async writes I created two partitions on a disk and created ext3 file systems on both the partitions. Also created two cgroups and generated lots of write traffic in two cgroups (50 fio threads) and watched the disk time statistics in respective cgroups at the interval of 2 seconds. Thanks to ryo tsuruta for the test case.
*****************************************************************
sync
echo 3 >/proc/sys/vm/drop_cachesfio_args="--size=64m --rw=write --numjobs=50 --group_reporting"
echo $$ > /cgroup/bfqio/test1/tasks
fio $fio_args --name=test1 --directory=/mnt/sdd1/fio/ --output=/mnt/sdd1/fio/test1.log &echo $$ > /cgroup/bfqio/test2/tasks
fio $fio_args --name=test2 --directory=/mnt/sdd2/fio/ --output=/mnt/sdd2/fio/test2.log &
***********************************************************************And watched the disk time and sector statistics for the both the cgroups every 2 seconds using a script. How is snippet from output.
test1 statistics: time=8 48 1315 sectors=8 48 55776 dq=8 48 1
test2 statistics: time=8 48 633 sectors=8 48 14720 dq=8 48 2test1 statistics: time=8 48 5586 sectors=8 48 339064 dq=8 48 2
test2 statistics: time=8 48 2985 sectors=8 48 146656 dq=8 48 3test1 statistics: time=8 48 9935 sectors=8 48 628728 dq=8 48 3
test2 statistics: time=8 48 5265 sectors=8 48 278688 dq=8 48 4test1 statistics: time=8 48 14156 sectors=8 48 932488 dq=8 48 6
test2 statistics: time=8 48 7646 sectors=8 48 412704 dq=8 48 7test1 statistics: time=8 48 18141 sectors=8 48 1231488 dq=8 48 10
test2 statistics: time=8 48 9820 sectors=8 48 548400 dq=8 48 8test1 statistics: time=8 48 21953 sectors=8 48 1485632 dq=8 48 13
test2 statistics: time=8 48 12394 sectors=8 48 698288 dq=8 48 10test1 statistics: time=8 48 25167 sectors=8 48 1705264 dq=8 48 13
test2 statistics: time=8 48 14042 sectors=8 48 817808 dq=8 48 10First two fields in time and sectors statistics represent major and minor number of the device. Third field represents disk time in milliseconds and number of sectors transferred respectively.
So disk time consumed by group1 is almost double of group2 in this case.
Your feedback is welcome.
Thanks
VivekSignature · 2 lines
Regards Gui Jianfeng
Quoted 10 lines · Vivek Goyal
Quoted 9 lines · Gui Jianfeng
Hi Vivek,
Here are some test results for normal reads and write for IO Controller V7 by fio. Tested with "fairness == 0". It seems performance gets better comparing with V6.
Mode Normal read | Random read | Normal write | Random write | Direct read | Direct Write
2.6.31-rc1 71,613KiB/s 3,606KiB/s 66,250KiB/s 9,420KiB/s 51,535KiB/s 55,752KiB/s
Quoted 7 lines
Quoted 4 lines
V7 70,540KiB/s 3,551KiB/s 64,548KiB/s 9,677KiB/s 53,530KiB/s 54,145KiB/s
Performance -1.5% -1.5% -2.6% +2.7% +3.9% -2.9%
Thanks Gui. Can you also try V7 with
CONFIG_TRACK_ASYNC_CONTEXT=n. I tried that and I got better results for buffered writes.
Yes, I'm also going to try it.
Quoted 419 lines
In my testing I still see some performance regression for buffered writes which goes away if I disable group io scheduling and just use flat mode.
I will spend more time to find out where it is coming from.
Thanks
VivekQuoted 406 lines
Quoted 402 lines · Vivek Goyal
Hi All,
Here is the V7 of the IO controller patches generated on top of 2.6.31-rc4.
For ease of patching, a consolidated patch is available here.
http://people.redhat.com/~vgoyal/io-controller/io-scheduler-based-io-controller-v7.patch
Previous versions of the patches was posted here.
(V1) http://lkml.org/lkml/2009/3/11/486
(V2) http://lkml.org/lkml/2009/5/5/275
(V3) http://lkml.org/lkml/2009/5/26/472
(V4) http://lkml.org/lkml/2009/6/8/580
(V5) http://lkml.org/lkml/2009/6/19/279
(V6) http://lkml.org/lkml/2009/7/2/369Changes from V6
===============
- Introduced the notion ofgroup_idlingwhere we idle for next request to come from the same group before we expire it. It is along the lines of cfq'sslice_idlething to provide fairness. Switching to group idling now helps in the sense that we don't have to rely whether queue idling was turned on or not by CFQ. It becomes too much of debugging pain with different work loads and different kind of storage media. Introduction ofgroup_idleshould help.- Moved some of the code like dynamic queue idling update, arming queue idling timer, keeping track of average think time etc back to CFQ. With group idling we don't need it now. Reduce the amount of change.
- Enabled cfq's close cooperator functionality in groups. So far this worked only in root group. Now it should work in non-root groups also.
- Got rid of the patch where we calculated disk time based on average disk rate in some circumstances. It was giving bad numbers in early queue deletion cases. Also did not think that it was helping a lot. Remvoed it for the time being.
- Added an experimental patch to map sync requests using bio tracking info and not task context. This is only for noop, deadline and AS.
- Got rid of experimental patch of idling for async queues. Don't think it was helping.
- Got rid of
wait_busyandwait_busy_donelogic from queue. Instead implemented it for groups.- Introduced
oom_ioqto accomodateoom_cfqqchange recently.- Broke-up
elv_init_ioq()fn into smaller functions. It had 7 arguments and looked complicated.- Fixed a bug in
blk_queue_io_group_congested(). Thanks to Munehiro Ikeda.- Merged gui's patch to fix the cgroup file format issue.
- Merged gui's patch to update per group congestion limit when
q->nr_group_requestsis updated.- Fixed a bug where close cooperation will not work if we wait for all the requests to finish from previous queue.
- Fixed group deletion accouting where deletion from idle tree were also appearing in the log.
- Got rid of
busy_rt_queuesinfrastructure.- Got rid of
elv_ioq_request_dispatched(). An helper function just to increment a variable.Limitations
===========- This IO controller provides the bandwidth control at the IO scheduler level (leaf node in stacked hiearchy of logical devices). So there can be cases (depending on configuration) where application does not see proportional BW division at higher logical level device.
LWN has written an article about the issue here.
http://lwn.net/Articles/332839/How to solve the issue of fairness at higher level logical devices
==================================================================
(Do we really need it? That's not where the contention for resources is.)Couple of suggestions have come forward.
- Implement IO control at IO scheduler layer and then with the help of some daemon, adjust the weight on underlying devices dynamiclly, depending on what kind of BW gurantees are to be achieved at higher level logical block devices.
- Also implement a higher level IO controller along with IO scheduler based controller and let user choose one depending on his needs.
A higher level controller does not know about the assumptions/policies of unerldying IO scheduler, hence it has the potential to break down the IO scheduler's policy with-in cgroup. A lower level controller can work with IO scheduler much more closely and efficiently.
Other active IO controller developments
=======================================IO throttling
-------------This is a max bandwidth controller and not the proportional one. Secondly it is a second level controller which can break the IO scheduler's policy/assumtions with-in cgroup.
dm-ioband
---------This is a proportional bandwidth controller implemented as device mapper driver. It is also a second level controller which can break the IO scheduler's policy/assumptions with-in cgroup.
TODO
====
- code cleanups, testing, bug fixing, optimizations, benchmarking etc...Testing
=======I have been able to do some testing as follows. All my testing is with ext3 file system with a SATA drive which supports queue depth of 31.
Test1 (Isolation between two KVM virtual machines)
==================================================
Created two KVM virtual machines. Partitioned a disk on host in two partitions and gave one partition to each virtual machine. Put both the virtual machines in two different cgroup of weight 1000 and 500 each. Virtual machines created ext3 file system on the partitions exported from host and did buffered writes. Host seems writes as synchronous and virtual machine with higher weight gets double the disk time of virtual machine of lower weight. Used deadline scheduler in this test case.Some more details about configuration are in documentation patch.
Test2 (Fairness for synchronous reads)
======================================
- Two dd in two cgroups with cgrop weights 1000 and 500. Ran two "dd" in those cgroups (With CFQ scheduler and /sys/block/<device>/queue/fairness = 1)Higher weight dd finishes first and at that point of time my script takes care of reading cgroup files io.disk_time and io.disk_sectors for both the groups and display the results.
dd if=/mnt/$BLOCKDEV/zerofile1 of=/dev/null &
dd if=/mnt/$BLOCKDEV/zerofile2 of=/dev/null &234179072 bytes (234 MB) copied, 3.9065 s, 59.9 MB/s 234179072 bytes (234 MB) copied, 5.19232 s, 45.1 MB/s
group1 time=8 16 2471 group1 sectors=8 16 457840 group2 time=8 16 1220 group2 sectors=8 16 225736
First two fields in time and sectors statistics represent major and minor number of the device. Third field represents disk time in milliseconds and number of sectors transferred respectively.
This patchset tries to provide fairness in terms of disk time received. group1 got almost double of group2 disk time (At the time of first dd finish). These time and sectors statistics can be read using io.disk_time and io.disk_sector files in cgroup. More about it in documentation file.
Test3 (Reader Vs Buffered Writes)
================================
Buffered writes can be problematic and can overwhelm readers, especially with noop and deadline. IO controller can provide isolation between readers and buffered (async) writers.First I ran the test without io controller to see the severity of the issue. Ran a hostile writer and then after 10 seconds started a reader and then monitored the completion time of reader. Reader reads a 256 MB file. Tested this with noop scheduler.
sample script
------------
sync
echo 3 >/proc/sys/vm/drop_caches
time dd if=/dev/zero of=/mnt/sdb/reader-writer-zerofile bs=4K count=2097152 conv=fdatasync &
sleep 10
time dd if=/mnt/sdb/256M-file of=/dev/null &Results
-------
8589934592 bytes (8.6 GB) copied, 106.045 s, 81.0 MB/s (Writer) 268435456 bytes (268 MB) copied, 96.5237 s, 2.8 MB/s (Reader)Now it was time to test io controller whether it can provide isolation between readers and writers with noop. I created two cgroups of weight 1000 each and put reader in group1 and writer in group 2 and ran the test again. Upon comletion of reader, my scripts read io.dis_time and io.disk_group cgroup files to get an estimate how much disk time each group got and how many sectors each group did IO for.
For more accurate accounting of disk time for buffered writes with queuing hardware I had to set /sys/block/<disk>/queue/iosched/fairness to "1".
sample script
-------------
echo $$ > /cgroup/bfqio/test2/tasks
dd if=/dev/zero of=/mnt/$BLOCKDEV/testzerofile bs=4K count=2097152 & sleep 10
echo noop >/sys/block/$BLOCKDEV/queue/scheduler
echo 1 >/sys/block/$BLOCKDEV/queue/iosched/fairnessecho $$ > /cgroup/bfqio/test1/tasks
dd if=/mnt/$BLOCKDEV/256M-file of=/dev/null &
wait $!
# Some code for reading cgroup files upon completion of reader.
-------------------------Results
=======
268435456 bytes (268 MB) copied, 6.65819 s, 40.3 MB/s (Reader)group1 time=8 16 3063 group1 sectors=8 16 524808
group2 time=8 16 3071 group2 sectors=8 16 441752Note, reader finishes now much lesser time and both group1 and group2 got almost 3 seconds of disk time. Hence io-controller provides isolation from buffered writes.
Test4 (AIO)
===========AIO reads
-----------
Set up two fio, AIO read jobs in two cgroup with weight 1000 and 500 respectively. I am using cfq scheduler. Following are some lines from my test script.---------------------------------------------------------------
echo 1000 > /cgroup/bfqio/test1/io.weight
echo 500 > /cgroup/bfqio/test2/io.weightfio_args="--ioengine=libaio --rw=read --size=512M --direct=1" echo 1 >
/sys/block/$BLOCKDEV/queue/iosched/fairnessecho $$ > /cgroup/bfqio/test1/tasks
fio $fio_args --name=test1 --directory=/mnt/$BLOCKDEV/fio1/ --output=/mnt/$BLOCKDEV/fio1/test1.log --exec_postrun="../read-and-display-group-stats.sh $maj_dev $minor_dev" &echo $$ > /cgroup/bfqio/test2/tasks
fio $fio_args --name=test2 --directory=/mnt/$BLOCKDEV/fio2/ --output=/mnt/$BLOCKDEV/fio2/test2.log &
----------------------------------------------------------------test1 and test2 are two groups with weight 1000 and 500 respectively. "
read-and-display-group-stats.sh" is one small script which reads the test1 and test2 cgroup files to determine how much disk time each group got till first fio job finished.Results
------
test1 statistics: time=8 16 22403 sectors=8 16 1049640
test2 statistics: time=8 16 11400 sectors=8 16 552864Above shows that by the time first fio (higher weight), finished, group test1 got 22403 ms of disk time and group test2 got 11400 ms of disk time. similarly the statistics for number of sectors transferred are also shown.
Note that disk time given to group test1 is almost double of group2 disk time.
AIO writes
----------
Set up two fio, AIO direct write jobs in two cgroup with weight 1000 and 500 respectively. I am using cfq scheduler. Following are some lines from my test script.------------------------------------------------
echo 1000 > /cgroup/bfqio/test1/io.weight
echo 500 > /cgroup/bfqio/test2/io.weight fio_args="--ioengine=libaio --rw=write --size=512M --direct=1"echo 1 >
/sys/block/$BLOCKDEV/queue/iosched/fairnessecho $$ > /cgroup/bfqio/test1/tasks
fio $fio_args --name=test1 --directory=/mnt/$BLOCKDEV/fio1/ --output=/mnt/$BLOCKDEV/fio1/test1.log --exec_postrun="../read-and-display-group-stats.sh $maj_dev $minor_dev" &echo $$ > /cgroup/bfqio/test2/tasks
fio $fio_args --name=test2 --directory=/mnt/$BLOCKDEV/fio2/ --output=/mnt/$BLOCKDEV/fio2/test2.log &
-------------------------------------------------test1 and test2 are two groups with weight 1000 and 500 respectively. "
read-and-display-group-stats.sh" is one small script which reads the test1 and test2 cgroup files to determine how much disk time each group got till first fio job finished.Following are the results.
test1 statistics: time=8 16 29085 sectors=8 16 1049656
test2 statistics: time=8 16 14652 sectors=8 16 516728Above shows that by the time first fio (higher weight), finished, group test1 got 28085 ms of disk time and group test2 got 14652 ms of disk time. similarly the statistics for number of sectors transferred are also shown.
Note that disk time given to group test1 is almost double of group2 disk time.
Test5 (Fairness for async writes, Buffered Write Vs Buffered Write)
===================================================================
Fairness for async writes is tricky and biggest reason is that async writes are cached in higher layers (page cahe) as well as possibly in file system layer also (btrfs, xfs etc), and are dispatched to lower layers not necessarily in proportional manner.For example, consider two dd threads reading
/dev/zeroas input file and doing writes of huge files. Very soon we will crossvm_dirty_ratioand dd thread will be forced to write out some pages to disk before more pages can be dirtied. But not necessarily dirty pages of same thread are picked. It can very well pick the inode of lesser priority dd thread and do some writeout. So effectively higher weight dd is doing writeouts of lower weight dd pages and we don't see service differentation.IOW, the core problem with async write fairness is that higher weight thread does not throw enought IO traffic at IO controller to keep the queue continuously backlogged. In my testing, there are many .2 to .8 second intervals where higher weight queue is empty and in that duration lower weight queue get lots of job done giving the impression that there was no service differentiation.
In summary, from IO controller point of view async writes support is there. Because page cache has not been designed in such a manner that higher prio/weight writer can do more write out as compared to lower prio/weight writer, gettting service differentiation is hard and it is visible in some cases and not visible in some cases.
Do we really care that much for fairness among two writer cgroups? One can choose to do direct writes or sync writes if fairness for writes really matters for him.
Following is the only case where it is hard to ensure fairness between cgroups.
- Buffered writes Vs Buffered Writes.
So to test async writes I created two partitions on a disk and created ext3 file systems on both the partitions. Also created two cgroups and generated lots of write traffic in two cgroups (50 fio threads) and watched the disk time statistics in respective cgroups at the interval of 2 seconds. Thanks to ryo tsuruta for the test case.
*****************************************************************
sync
echo 3 >/proc/sys/vm/drop_cachesfio_args="--size=64m --rw=write --numjobs=50 --group_reporting"
echo $$ > /cgroup/bfqio/test1/tasks
fio $fio_args --name=test1 --directory=/mnt/sdd1/fio/ --output=/mnt/sdd1/fio/test1.log &echo $$ > /cgroup/bfqio/test2/tasks
fio $fio_args --name=test2 --directory=/mnt/sdd2/fio/ --output=/mnt/sdd2/fio/test2.log &
***********************************************************************And watched the disk time and sector statistics for the both the cgroups every 2 seconds using a script. How is snippet from output.
test1 statistics: time=8 48 1315 sectors=8 48 55776 dq=8 48 1
test2 statistics: time=8 48 633 sectors=8 48 14720 dq=8 48 2test1 statistics: time=8 48 5586 sectors=8 48 339064 dq=8 48 2
test2 statistics: time=8 48 2985 sectors=8 48 146656 dq=8 48 3test1 statistics: time=8 48 9935 sectors=8 48 628728 dq=8 48 3
test2 statistics: time=8 48 5265 sectors=8 48 278688 dq=8 48 4test1 statistics: time=8 48 14156 sectors=8 48 932488 dq=8 48 6
test2 statistics: time=8 48 7646 sectors=8 48 412704 dq=8 48 7test1 statistics: time=8 48 18141 sectors=8 48 1231488 dq=8 48 10
test2 statistics: time=8 48 9820 sectors=8 48 548400 dq=8 48 8test1 statistics: time=8 48 21953 sectors=8 48 1485632 dq=8 48 13
test2 statistics: time=8 48 12394 sectors=8 48 698288 dq=8 48 10test1 statistics: time=8 48 25167 sectors=8 48 1705264 dq=8 48 13
test2 statistics: time=8 48 14042 sectors=8 48 817808 dq=8 48 10First two fields in time and sectors statistics represent major and minor number of the device. Third field represents disk time in milliseconds and number of sectors transferred respectively.
So disk time consumed by group1 is almost double of group2 in this case.
Your feedback is welcome.
Thanks
VivekSignature · 2 lines
Regards Gui Jianfeng
Signature · 2 lines
Regards Gui Jianfeng
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 10 lines · Vivek Goyal
Quoted 9 lines · Gui Jianfeng
Hi Vivek,
Here are some test results for normal reads and write for IO Controller V7 by fio. Tested with "fairness == 0". It seems performance gets better comparing with V6.
Mode Normal read | Random read | Normal write | Random write | Direct read | Direct Write
2.6.31-rc1 71,613KiB/s 3,606KiB/s 66,250KiB/s 9,420KiB/s 51,535KiB/s 55,752KiB/s
Quoted 7 lines
Quoted 4 lines
V7 70,540KiB/s 3,551KiB/s 64,548KiB/s 9,677KiB/s 53,530KiB/s 54,145KiB/s
Performance -1.5% -1.5% -2.6% +2.7% +3.9% -2.9%
Thanks Gui. Can you also try V7 with
CONFIG_TRACK_ASYNC_CONTEXT=n. I tried that and I got better results for buffered writes.
Yes, I'm also going to try it.
Quoted 419 lines
In my testing I still see some performance regression for buffered writes which goes away if I disable group io scheduling and just use flat mode.
I will spend more time to find out where it is coming from.
Thanks
VivekQuoted 406 lines
Quoted 402 lines · Vivek Goyal
Hi All,
Here is the V7 of the IO controller patches generated on top of 2.6.31-rc4.
For ease of patching, a consolidated patch is available here.
http://people.redhat.com/~vgoyal/io-controller/io-scheduler-based-io-controller-v7.patch
Previous versions of the patches was posted here.
(V1) http://lkml.org/lkml/2009/3/11/486
(V2) http://lkml.org/lkml/2009/5/5/275
(V3) http://lkml.org/lkml/2009/5/26/472
(V4) http://lkml.org/lkml/2009/6/8/580
(V5) http://lkml.org/lkml/2009/6/19/279
(V6) http://lkml.org/lkml/2009/7/2/369Changes from V6
===============
- Introduced the notion ofgroup_idlingwhere we idle for next request to come from the same group before we expire it. It is along the lines of cfq'sslice_idlething to provide fairness. Switching to group idling now helps in the sense that we don't have to rely whether queue idling was turned on or not by CFQ. It becomes too much of debugging pain with different work loads and different kind of storage media. Introduction ofgroup_idleshould help.- Moved some of the code like dynamic queue idling update, arming queue idling timer, keeping track of average think time etc back to CFQ. With group idling we don't need it now. Reduce the amount of change.
- Enabled cfq's close cooperator functionality in groups. So far this worked only in root group. Now it should work in non-root groups also.
- Got rid of the patch where we calculated disk time based on average disk rate in some circumstances. It was giving bad numbers in early queue deletion cases. Also did not think that it was helping a lot. Remvoed it for the time being.
- Added an experimental patch to map sync requests using bio tracking info and not task context. This is only for noop, deadline and AS.
- Got rid of experimental patch of idling for async queues. Don't think it was helping.
- Got rid of
wait_busyandwait_busy_donelogic from queue. Instead implemented it for groups.- Introduced
oom_ioqto accomodateoom_cfqqchange recently.- Broke-up
elv_init_ioq()fn into smaller functions. It had 7 arguments and looked complicated.- Fixed a bug in
blk_queue_io_group_congested(). Thanks to Munehiro Ikeda.- Merged gui's patch to fix the cgroup file format issue.
- Merged gui's patch to update per group congestion limit when
q->nr_group_requestsis updated.- Fixed a bug where close cooperation will not work if we wait for all the requests to finish from previous queue.
- Fixed group deletion accouting where deletion from idle tree were also appearing in the log.
- Got rid of
busy_rt_queuesinfrastructure.- Got rid of
elv_ioq_request_dispatched(). An helper function just to increment a variable.Limitations
===========- This IO controller provides the bandwidth control at the IO scheduler level (leaf node in stacked hiearchy of logical devices). So there can be cases (depending on configuration) where application does not see proportional BW division at higher logical level device.
LWN has written an article about the issue here.
http://lwn.net/Articles/332839/How to solve the issue of fairness at higher level logical devices
==================================================================
(Do we really need it? That's not where the contention for resources is.)Couple of suggestions have come forward.
- Implement IO control at IO scheduler layer and then with the help of some daemon, adjust the weight on underlying devices dynamiclly, depending on what kind of BW gurantees are to be achieved at higher level logical block devices.
- Also implement a higher level IO controller along with IO scheduler based controller and let user choose one depending on his needs.
A higher level controller does not know about the assumptions/policies of unerldying IO scheduler, hence it has the potential to break down the IO scheduler's policy with-in cgroup. A lower level controller can work with IO scheduler much more closely and efficiently.
Other active IO controller developments
=======================================IO throttling
-------------This is a max bandwidth controller and not the proportional one. Secondly it is a second level controller which can break the IO scheduler's policy/assumtions with-in cgroup.
dm-ioband
---------This is a proportional bandwidth controller implemented as device mapper driver. It is also a second level controller which can break the IO scheduler's policy/assumptions with-in cgroup.
TODO
====
- code cleanups, testing, bug fixing, optimizations, benchmarking etc...Testing
=======I have been able to do some testing as follows. All my testing is with ext3 file system with a SATA drive which supports queue depth of 31.
Test1 (Isolation between two KVM virtual machines)
==================================================
Created two KVM virtual machines. Partitioned a disk on host in two partitions and gave one partition to each virtual machine. Put both the virtual machines in two different cgroup of weight 1000 and 500 each. Virtual machines created ext3 file system on the partitions exported from host and did buffered writes. Host seems writes as synchronous and virtual machine with higher weight gets double the disk time of virtual machine of lower weight. Used deadline scheduler in this test case.Some more details about configuration are in documentation patch.
Test2 (Fairness for synchronous reads)
======================================
- Two dd in two cgroups with cgrop weights 1000 and 500. Ran two "dd" in those cgroups (With CFQ scheduler and /sys/block/<device>/queue/fairness = 1)Higher weight dd finishes first and at that point of time my script takes care of reading cgroup files io.disk_time and io.disk_sectors for both the groups and display the results.
dd if=/mnt/$BLOCKDEV/zerofile1 of=/dev/null &
dd if=/mnt/$BLOCKDEV/zerofile2 of=/dev/null &234179072 bytes (234 MB) copied, 3.9065 s, 59.9 MB/s 234179072 bytes (234 MB) copied, 5.19232 s, 45.1 MB/s
group1 time=8 16 2471 group1 sectors=8 16 457840 group2 time=8 16 1220 group2 sectors=8 16 225736
First two fields in time and sectors statistics represent major and minor number of the device. Third field represents disk time in milliseconds and number of sectors transferred respectively.
This patchset tries to provide fairness in terms of disk time received. group1 got almost double of group2 disk time (At the time of first dd finish). These time and sectors statistics can be read using io.disk_time and io.disk_sector files in cgroup. More about it in documentation file.
Test3 (Reader Vs Buffered Writes)
================================
Buffered writes can be problematic and can overwhelm readers, especially with noop and deadline. IO controller can provide isolation between readers and buffered (async) writers.First I ran the test without io controller to see the severity of the issue. Ran a hostile writer and then after 10 seconds started a reader and then monitored the completion time of reader. Reader reads a 256 MB file. Tested this with noop scheduler.
sample script
------------
sync
echo 3 >/proc/sys/vm/drop_caches
time dd if=/dev/zero of=/mnt/sdb/reader-writer-zerofile bs=4K count=2097152 conv=fdatasync &
sleep 10
time dd if=/mnt/sdb/256M-file of=/dev/null &Results
-------
8589934592 bytes (8.6 GB) copied, 106.045 s, 81.0 MB/s (Writer) 268435456 bytes (268 MB) copied, 96.5237 s, 2.8 MB/s (Reader)Now it was time to test io controller whether it can provide isolation between readers and writers with noop. I created two cgroups of weight 1000 each and put reader in group1 and writer in group 2 and ran the test again. Upon comletion of reader, my scripts read io.dis_time and io.disk_group cgroup files to get an estimate how much disk time each group got and how many sectors each group did IO for.
For more accurate accounting of disk time for buffered writes with queuing hardware I had to set /sys/block/<disk>/queue/iosched/fairness to "1".
sample script
-------------
echo $$ > /cgroup/bfqio/test2/tasks
dd if=/dev/zero of=/mnt/$BLOCKDEV/testzerofile bs=4K count=2097152 & sleep 10
echo noop >/sys/block/$BLOCKDEV/queue/scheduler
echo 1 >/sys/block/$BLOCKDEV/queue/iosched/fairnessecho $$ > /cgroup/bfqio/test1/tasks
dd if=/mnt/$BLOCKDEV/256M-file of=/dev/null &
wait $!
# Some code for reading cgroup files upon completion of reader.
-------------------------Results
=======
268435456 bytes (268 MB) copied, 6.65819 s, 40.3 MB/s (Reader)group1 time=8 16 3063 group1 sectors=8 16 524808
group2 time=8 16 3071 group2 sectors=8 16 441752Note, reader finishes now much lesser time and both group1 and group2 got almost 3 seconds of disk time. Hence io-controller provides isolation from buffered writes.
Test4 (AIO)
===========AIO reads
-----------
Set up two fio, AIO read jobs in two cgroup with weight 1000 and 500 respectively. I am using cfq scheduler. Following are some lines from my test script.---------------------------------------------------------------
echo 1000 > /cgroup/bfqio/test1/io.weight
echo 500 > /cgroup/bfqio/test2/io.weightfio_args="--ioengine=libaio --rw=read --size=512M --direct=1" echo 1 >
/sys/block/$BLOCKDEV/queue/iosched/fairnessecho $$ > /cgroup/bfqio/test1/tasks
fio $fio_args --name=test1 --directory=/mnt/$BLOCKDEV/fio1/ --output=/mnt/$BLOCKDEV/fio1/test1.log --exec_postrun="../read-and-display-group-stats.sh $maj_dev $minor_dev" &echo $$ > /cgroup/bfqio/test2/tasks
fio $fio_args --name=test2 --directory=/mnt/$BLOCKDEV/fio2/ --output=/mnt/$BLOCKDEV/fio2/test2.log &
----------------------------------------------------------------test1 and test2 are two groups with weight 1000 and 500 respectively. "
read-and-display-group-stats.sh" is one small script which reads the test1 and test2 cgroup files to determine how much disk time each group got till first fio job finished.Results
------
test1 statistics: time=8 16 22403 sectors=8 16 1049640
test2 statistics: time=8 16 11400 sectors=8 16 552864Above shows that by the time first fio (higher weight), finished, group test1 got 22403 ms of disk time and group test2 got 11400 ms of disk time. similarly the statistics for number of sectors transferred are also shown.
Note that disk time given to group test1 is almost double of group2 disk time.
AIO writes
----------
Set up two fio, AIO direct write jobs in two cgroup with weight 1000 and 500 respectively. I am using cfq scheduler. Following are some lines from my test script.------------------------------------------------
echo 1000 > /cgroup/bfqio/test1/io.weight
echo 500 > /cgroup/bfqio/test2/io.weight fio_args="--ioengine=libaio --rw=write --size=512M --direct=1"echo 1 >
/sys/block/$BLOCKDEV/queue/iosched/fairnessecho $$ > /cgroup/bfqio/test1/tasks
fio $fio_args --name=test1 --directory=/mnt/$BLOCKDEV/fio1/ --output=/mnt/$BLOCKDEV/fio1/test1.log --exec_postrun="../read-and-display-group-stats.sh $maj_dev $minor_dev" &echo $$ > /cgroup/bfqio/test2/tasks
fio $fio_args --name=test2 --directory=/mnt/$BLOCKDEV/fio2/ --output=/mnt/$BLOCKDEV/fio2/test2.log &
-------------------------------------------------test1 and test2 are two groups with weight 1000 and 500 respectively. "
read-and-display-group-stats.sh" is one small script which reads the test1 and test2 cgroup files to determine how much disk time each group got till first fio job finished.Following are the results.
test1 statistics: time=8 16 29085 sectors=8 16 1049656
test2 statistics: time=8 16 14652 sectors=8 16 516728Above shows that by the time first fio (higher weight), finished, group test1 got 28085 ms of disk time and group test2 got 14652 ms of disk time. similarly the statistics for number of sectors transferred are also shown.
Note that disk time given to group test1 is almost double of group2 disk time.
Test5 (Fairness for async writes, Buffered Write Vs Buffered Write)
===================================================================
Fairness for async writes is tricky and biggest reason is that async writes are cached in higher layers (page cahe) as well as possibly in file system layer also (btrfs, xfs etc), and are dispatched to lower layers not necessarily in proportional manner.For example, consider two dd threads reading
/dev/zeroas input file and doing writes of huge files. Very soon we will crossvm_dirty_ratioand dd thread will be forced to write out some pages to disk before more pages can be dirtied. But not necessarily dirty pages of same thread are picked. It can very well pick the inode of lesser priority dd thread and do some writeout. So effectively higher weight dd is doing writeouts of lower weight dd pages and we don't see service differentation.IOW, the core problem with async write fairness is that higher weight thread does not throw enought IO traffic at IO controller to keep the queue continuously backlogged. In my testing, there are many .2 to .8 second intervals where higher weight queue is empty and in that duration lower weight queue get lots of job done giving the impression that there was no service differentiation.
In summary, from IO controller point of view async writes support is there. Because page cache has not been designed in such a manner that higher prio/weight writer can do more write out as compared to lower prio/weight writer, gettting service differentiation is hard and it is visible in some cases and not visible in some cases.
Do we really care that much for fairness among two writer cgroups? One can choose to do direct writes or sync writes if fairness for writes really matters for him.
Following is the only case where it is hard to ensure fairness between cgroups.
- Buffered writes Vs Buffered Writes.
So to test async writes I created two partitions on a disk and created ext3 file systems on both the partitions. Also created two cgroups and generated lots of write traffic in two cgroups (50 fio threads) and watched the disk time statistics in respective cgroups at the interval of 2 seconds. Thanks to ryo tsuruta for the test case.
*****************************************************************
sync
echo 3 >/proc/sys/vm/drop_cachesfio_args="--size=64m --rw=write --numjobs=50 --group_reporting"
echo $$ > /cgroup/bfqio/test1/tasks
fio $fio_args --name=test1 --directory=/mnt/sdd1/fio/ --output=/mnt/sdd1/fio/test1.log &echo $$ > /cgroup/bfqio/test2/tasks
fio $fio_args --name=test2 --directory=/mnt/sdd2/fio/ --output=/mnt/sdd2/fio/test2.log &
***********************************************************************And watched the disk time and sector statistics for the both the cgroups every 2 seconds using a script. How is snippet from output.
test1 statistics: time=8 48 1315 sectors=8 48 55776 dq=8 48 1
test2 statistics: time=8 48 633 sectors=8 48 14720 dq=8 48 2test1 statistics: time=8 48 5586 sectors=8 48 339064 dq=8 48 2
test2 statistics: time=8 48 2985 sectors=8 48 146656 dq=8 48 3test1 statistics: time=8 48 9935 sectors=8 48 628728 dq=8 48 3
test2 statistics: time=8 48 5265 sectors=8 48 278688 dq=8 48 4test1 statistics: time=8 48 14156 sectors=8 48 932488 dq=8 48 6
test2 statistics: time=8 48 7646 sectors=8 48 412704 dq=8 48 7test1 statistics: time=8 48 18141 sectors=8 48 1231488 dq=8 48 10
test2 statistics: time=8 48 9820 sectors=8 48 548400 dq=8 48 8test1 statistics: time=8 48 21953 sectors=8 48 1485632 dq=8 48 13
test2 statistics: time=8 48 12394 sectors=8 48 698288 dq=8 48 10test1 statistics: time=8 48 25167 sectors=8 48 1705264 dq=8 48 13
test2 statistics: time=8 48 14042 sectors=8 48 817808 dq=8 48 10First two fields in time and sectors statistics represent major and minor number of the device. Third field represents disk time in milliseconds and number of sectors transferred respectively.
So disk time consumed by group1 is almost double of group2 in this case.
Your feedback is welcome.
Thanks
VivekSignature · 2 lines
Regards Gui Jianfeng
Signature · 2 lines
Regards Gui Jianfeng
Vivek, Here are some test results with and without CONFIG_TRACK_ASYNC_CONTEXT for V7
Mode Normal read | Random read | Normal write | Random write | Direct read | Direct Write
CONFIG_TRACK_ASYNC_CONTEXT=y 70,540KiB/s 3,551KiB/s 64,548KiB/s 9,677KiB/s 53,530KiB/s 54,145KiB/s
CONFIG_TRACK_ASYNC_CONTEXT=n 71,082KiB/s 3,564KiB/s 66,720KiB/s 9,887KiB/s 51,401KiB/s 55,210KiB/s
Performance +0.7% +0.3% +3.3% +2.1% -4.0% +2.0%
Quoted 436 lines · Vivek Goyal
Quoted 5 lines · Gui Jianfeng
Hi Vivek,
Here are some test results for normal reads and write for IO Controller V7 by fio. Tested with "fairness == 0". It seems performance gets better comparing with V6.
Quoted 8 lines
Mode Normal read | Random read | Normal write | Random write | Direct read | Direct Write
2.6.31-rc1 71,613KiB/s 3,606KiB/s 66,250KiB/s 9,420KiB/s 51,535KiB/s 55,752KiB/s
V7 70,540KiB/s 3,551KiB/s 64,548KiB/s 9,677KiB/s 53,530KiB/s 54,145KiB/s
Performance -1.5% -1.5% -2.6% +2.7% +3.9% -2.9%
Thanks Gui. Can you also try V7 with
CONFIG_TRACK_ASYNC_CONTEXT=n. I tried that and I got better results for buffered writes.In my testing I still see some performance regression for buffered writes which goes away if I disable group io scheduling and just use flat mode.
I will spend more time to find out where it is coming from.
Thanks
VivekQuoted 406 lines
Quoted 402 lines · Vivek Goyal
Hi All,
Here is the V7 of the IO controller patches generated on top of 2.6.31-rc4.
For ease of patching, a consolidated patch is available here.
http://people.redhat.com/~vgoyal/io-controller/io-scheduler-based-io-controller-v7.patch
Previous versions of the patches was posted here.
(V1) http://lkml.org/lkml/2009/3/11/486
(V2) http://lkml.org/lkml/2009/5/5/275
(V3) http://lkml.org/lkml/2009/5/26/472
(V4) http://lkml.org/lkml/2009/6/8/580
(V5) http://lkml.org/lkml/2009/6/19/279
(V6) http://lkml.org/lkml/2009/7/2/369Changes from V6
===============
- Introduced the notion ofgroup_idlingwhere we idle for next request to come from the same group before we expire it. It is along the lines of cfq'sslice_idlething to provide fairness. Switching to group idling now helps in the sense that we don't have to rely whether queue idling was turned on or not by CFQ. It becomes too much of debugging pain with different work loads and different kind of storage media. Introduction ofgroup_idleshould help.- Moved some of the code like dynamic queue idling update, arming queue idling timer, keeping track of average think time etc back to CFQ. With group idling we don't need it now. Reduce the amount of change.
- Enabled cfq's close cooperator functionality in groups. So far this worked only in root group. Now it should work in non-root groups also.
- Got rid of the patch where we calculated disk time based on average disk rate in some circumstances. It was giving bad numbers in early queue deletion cases. Also did not think that it was helping a lot. Remvoed it for the time being.
- Added an experimental patch to map sync requests using bio tracking info and not task context. This is only for noop, deadline and AS.
- Got rid of experimental patch of idling for async queues. Don't think it was helping.
- Got rid of
wait_busyandwait_busy_donelogic from queue. Instead implemented it for groups.- Introduced
oom_ioqto accomodateoom_cfqqchange recently.- Broke-up
elv_init_ioq()fn into smaller functions. It had 7 arguments and looked complicated.- Fixed a bug in
blk_queue_io_group_congested(). Thanks to Munehiro Ikeda.- Merged gui's patch to fix the cgroup file format issue.
- Merged gui's patch to update per group congestion limit when
q->nr_group_requestsis updated.- Fixed a bug where close cooperation will not work if we wait for all the requests to finish from previous queue.
- Fixed group deletion accouting where deletion from idle tree were also appearing in the log.
- Got rid of
busy_rt_queuesinfrastructure.- Got rid of
elv_ioq_request_dispatched(). An helper function just to increment a variable.Limitations
===========- This IO controller provides the bandwidth control at the IO scheduler level (leaf node in stacked hiearchy of logical devices). So there can be cases (depending on configuration) where application does not see proportional BW division at higher logical level device.
LWN has written an article about the issue here.
http://lwn.net/Articles/332839/How to solve the issue of fairness at higher level logical devices
==================================================================
(Do we really need it? That's not where the contention for resources is.)Couple of suggestions have come forward.
- Implement IO control at IO scheduler layer and then with the help of some daemon, adjust the weight on underlying devices dynamiclly, depending on what kind of BW gurantees are to be achieved at higher level logical block devices.
- Also implement a higher level IO controller along with IO scheduler based controller and let user choose one depending on his needs.
A higher level controller does not know about the assumptions/policies of unerldying IO scheduler, hence it has the potential to break down the IO scheduler's policy with-in cgroup. A lower level controller can work with IO scheduler much more closely and efficiently.
Other active IO controller developments
=======================================IO throttling
-------------This is a max bandwidth controller and not the proportional one. Secondly it is a second level controller which can break the IO scheduler's policy/assumtions with-in cgroup.
dm-ioband
---------This is a proportional bandwidth controller implemented as device mapper driver. It is also a second level controller which can break the IO scheduler's policy/assumptions with-in cgroup.
TODO
====
- code cleanups, testing, bug fixing, optimizations, benchmarking etc...Testing
=======I have been able to do some testing as follows. All my testing is with ext3 file system with a SATA drive which supports queue depth of 31.
Test1 (Isolation between two KVM virtual machines)
==================================================
Created two KVM virtual machines. Partitioned a disk on host in two partitions and gave one partition to each virtual machine. Put both the virtual machines in two different cgroup of weight 1000 and 500 each. Virtual machines created ext3 file system on the partitions exported from host and did buffered writes. Host seems writes as synchronous and virtual machine with higher weight gets double the disk time of virtual machine of lower weight. Used deadline scheduler in this test case.Some more details about configuration are in documentation patch.
Test2 (Fairness for synchronous reads)
======================================
- Two dd in two cgroups with cgrop weights 1000 and 500. Ran two "dd" in those cgroups (With CFQ scheduler and /sys/block/<device>/queue/fairness = 1)Higher weight dd finishes first and at that point of time my script takes care of reading cgroup files io.disk_time and io.disk_sectors for both the groups and display the results.
dd if=/mnt/$BLOCKDEV/zerofile1 of=/dev/null &
dd if=/mnt/$BLOCKDEV/zerofile2 of=/dev/null &234179072 bytes (234 MB) copied, 3.9065 s, 59.9 MB/s 234179072 bytes (234 MB) copied, 5.19232 s, 45.1 MB/s
group1 time=8 16 2471 group1 sectors=8 16 457840 group2 time=8 16 1220 group2 sectors=8 16 225736
First two fields in time and sectors statistics represent major and minor number of the device. Third field represents disk time in milliseconds and number of sectors transferred respectively.
This patchset tries to provide fairness in terms of disk time received. group1 got almost double of group2 disk time (At the time of first dd finish). These time and sectors statistics can be read using io.disk_time and io.disk_sector files in cgroup. More about it in documentation file.
Test3 (Reader Vs Buffered Writes)
================================
Buffered writes can be problematic and can overwhelm readers, especially with noop and deadline. IO controller can provide isolation between readers and buffered (async) writers.First I ran the test without io controller to see the severity of the issue. Ran a hostile writer and then after 10 seconds started a reader and then monitored the completion time of reader. Reader reads a 256 MB file. Tested this with noop scheduler.
sample script
------------
sync
echo 3 >/proc/sys/vm/drop_caches
time dd if=/dev/zero of=/mnt/sdb/reader-writer-zerofile bs=4K count=2097152 conv=fdatasync &
sleep 10
time dd if=/mnt/sdb/256M-file of=/dev/null &Results
-------
8589934592 bytes (8.6 GB) copied, 106.045 s, 81.0 MB/s (Writer) 268435456 bytes (268 MB) copied, 96.5237 s, 2.8 MB/s (Reader)Now it was time to test io controller whether it can provide isolation between readers and writers with noop. I created two cgroups of weight 1000 each and put reader in group1 and writer in group 2 and ran the test again. Upon comletion of reader, my scripts read io.dis_time and io.disk_group cgroup files to get an estimate how much disk time each group got and how many sectors each group did IO for.
For more accurate accounting of disk time for buffered writes with queuing hardware I had to set /sys/block/<disk>/queue/iosched/fairness to "1".
sample script
-------------
echo $$ > /cgroup/bfqio/test2/tasks
dd if=/dev/zero of=/mnt/$BLOCKDEV/testzerofile bs=4K count=2097152 & sleep 10
echo noop >/sys/block/$BLOCKDEV/queue/scheduler
echo 1 >/sys/block/$BLOCKDEV/queue/iosched/fairnessecho $$ > /cgroup/bfqio/test1/tasks
dd if=/mnt/$BLOCKDEV/256M-file of=/dev/null &
wait $!
# Some code for reading cgroup files upon completion of reader.
-------------------------Results
=======
268435456 bytes (268 MB) copied, 6.65819 s, 40.3 MB/s (Reader)group1 time=8 16 3063 group1 sectors=8 16 524808
group2 time=8 16 3071 group2 sectors=8 16 441752Note, reader finishes now much lesser time and both group1 and group2 got almost 3 seconds of disk time. Hence io-controller provides isolation from buffered writes.
Test4 (AIO)
===========AIO reads
-----------
Set up two fio, AIO read jobs in two cgroup with weight 1000 and 500 respectively. I am using cfq scheduler. Following are some lines from my test script.---------------------------------------------------------------
echo 1000 > /cgroup/bfqio/test1/io.weight
echo 500 > /cgroup/bfqio/test2/io.weightfio_args="--ioengine=libaio --rw=read --size=512M --direct=1" echo 1 >
/sys/block/$BLOCKDEV/queue/iosched/fairnessecho $$ > /cgroup/bfqio/test1/tasks
fio $fio_args --name=test1 --directory=/mnt/$BLOCKDEV/fio1/ --output=/mnt/$BLOCKDEV/fio1/test1.log --exec_postrun="../read-and-display-group-stats.sh $maj_dev $minor_dev" &echo $$ > /cgroup/bfqio/test2/tasks
fio $fio_args --name=test2 --directory=/mnt/$BLOCKDEV/fio2/ --output=/mnt/$BLOCKDEV/fio2/test2.log &
----------------------------------------------------------------test1 and test2 are two groups with weight 1000 and 500 respectively. "
read-and-display-group-stats.sh" is one small script which reads the test1 and test2 cgroup files to determine how much disk time each group got till first fio job finished.Results
------
test1 statistics: time=8 16 22403 sectors=8 16 1049640
test2 statistics: time=8 16 11400 sectors=8 16 552864Above shows that by the time first fio (higher weight), finished, group test1 got 22403 ms of disk time and group test2 got 11400 ms of disk time. similarly the statistics for number of sectors transferred are also shown.
Note that disk time given to group test1 is almost double of group2 disk time.
AIO writes
----------
Set up two fio, AIO direct write jobs in two cgroup with weight 1000 and 500 respectively. I am using cfq scheduler. Following are some lines from my test script.------------------------------------------------
echo 1000 > /cgroup/bfqio/test1/io.weight
echo 500 > /cgroup/bfqio/test2/io.weight fio_args="--ioengine=libaio --rw=write --size=512M --direct=1"echo 1 >
/sys/block/$BLOCKDEV/queue/iosched/fairnessecho $$ > /cgroup/bfqio/test1/tasks
fio $fio_args --name=test1 --directory=/mnt/$BLOCKDEV/fio1/ --output=/mnt/$BLOCKDEV/fio1/test1.log --exec_postrun="../read-and-display-group-stats.sh $maj_dev $minor_dev" &echo $$ > /cgroup/bfqio/test2/tasks
fio $fio_args --name=test2 --directory=/mnt/$BLOCKDEV/fio2/ --output=/mnt/$BLOCKDEV/fio2/test2.log &
-------------------------------------------------test1 and test2 are two groups with weight 1000 and 500 respectively. "
read-and-display-group-stats.sh" is one small script which reads the test1 and test2 cgroup files to determine how much disk time each group got till first fio job finished.Following are the results.
test1 statistics: time=8 16 29085 sectors=8 16 1049656
test2 statistics: time=8 16 14652 sectors=8 16 516728Above shows that by the time first fio (higher weight), finished, group test1 got 28085 ms of disk time and group test2 got 14652 ms of disk time. similarly the statistics for number of sectors transferred are also shown.
Note that disk time given to group test1 is almost double of group2 disk time.
Test5 (Fairness for async writes, Buffered Write Vs Buffered Write)
===================================================================
Fairness for async writes is tricky and biggest reason is that async writes are cached in higher layers (page cahe) as well as possibly in file system layer also (btrfs, xfs etc), and are dispatched to lower layers not necessarily in proportional manner.For example, consider two dd threads reading
/dev/zeroas input file and doing writes of huge files. Very soon we will crossvm_dirty_ratioand dd thread will be forced to write out some pages to disk before more pages can be dirtied. But not necessarily dirty pages of same thread are picked. It can very well pick the inode of lesser priority dd thread and do some writeout. So effectively higher weight dd is doing writeouts of lower weight dd pages and we don't see service differentation.IOW, the core problem with async write fairness is that higher weight thread does not throw enought IO traffic at IO controller to keep the queue continuously backlogged. In my testing, there are many .2 to .8 second intervals where higher weight queue is empty and in that duration lower weight queue get lots of job done giving the impression that there was no service differentiation.
In summary, from IO controller point of view async writes support is there. Because page cache has not been designed in such a manner that higher prio/weight writer can do more write out as compared to lower prio/weight writer, gettting service differentiation is hard and it is visible in some cases and not visible in some cases.
Do we really care that much for fairness among two writer cgroups? One can choose to do direct writes or sync writes if fairness for writes really matters for him.
Following is the only case where it is hard to ensure fairness between cgroups.
- Buffered writes Vs Buffered Writes.
So to test async writes I created two partitions on a disk and created ext3 file systems on both the partitions. Also created two cgroups and generated lots of write traffic in two cgroups (50 fio threads) and watched the disk time statistics in respective cgroups at the interval of 2 seconds. Thanks to ryo tsuruta for the test case.
*****************************************************************
sync
echo 3 >/proc/sys/vm/drop_cachesfio_args="--size=64m --rw=write --numjobs=50 --group_reporting"
echo $$ > /cgroup/bfqio/test1/tasks
fio $fio_args --name=test1 --directory=/mnt/sdd1/fio/ --output=/mnt/sdd1/fio/test1.log &echo $$ > /cgroup/bfqio/test2/tasks
fio $fio_args --name=test2 --directory=/mnt/sdd2/fio/ --output=/mnt/sdd2/fio/test2.log &
***********************************************************************And watched the disk time and sector statistics for the both the cgroups every 2 seconds using a script. How is snippet from output.
test1 statistics: time=8 48 1315 sectors=8 48 55776 dq=8 48 1
test2 statistics: time=8 48 633 sectors=8 48 14720 dq=8 48 2test1 statistics: time=8 48 5586 sectors=8 48 339064 dq=8 48 2
test2 statistics: time=8 48 2985 sectors=8 48 146656 dq=8 48 3test1 statistics: time=8 48 9935 sectors=8 48 628728 dq=8 48 3
test2 statistics: time=8 48 5265 sectors=8 48 278688 dq=8 48 4test1 statistics: time=8 48 14156 sectors=8 48 932488 dq=8 48 6
test2 statistics: time=8 48 7646 sectors=8 48 412704 dq=8 48 7test1 statistics: time=8 48 18141 sectors=8 48 1231488 dq=8 48 10
test2 statistics: time=8 48 9820 sectors=8 48 548400 dq=8 48 8test1 statistics: time=8 48 21953 sectors=8 48 1485632 dq=8 48 13
test2 statistics: time=8 48 12394 sectors=8 48 698288 dq=8 48 10test1 statistics: time=8 48 25167 sectors=8 48 1705264 dq=8 48 13
test2 statistics: time=8 48 14042 sectors=8 48 817808 dq=8 48 10First two fields in time and sectors statistics represent major and minor number of the device. Third field represents disk time in milliseconds and number of sectors transferred respectively.
So disk time consumed by group1 is almost double of group2 in this case.
Your feedback is welcome.
Thanks
VivekSignature · 2 lines
Regards Gui Jianfeng
Signature · 2 lines
Regards Gui Jianfeng
Vivek, Here are some test results with and without CONFIG_TRACK_ASYNC_CONTEXT for V7
Mode Normal read | Random read | Normal write | Random write | Direct read | Direct Write
CONFIG_TRACK_ASYNC_CONTEXT=y 70,540KiB/s 3,551KiB/s 64,548KiB/s 9,677KiB/s 53,530KiB/s 54,145KiB/s
CONFIG_TRACK_ASYNC_CONTEXT=n 71,082KiB/s 3,564KiB/s 66,720KiB/s 9,887KiB/s 51,401KiB/s 55,210KiB/s
Performance +0.7% +0.3% +3.3% +2.1% -4.0% +2.0%
Quoted 436 lines · Vivek Goyal
Quoted 5 lines · Gui Jianfeng
Hi Vivek,
Here are some test results for normal reads and write for IO Controller V7 by fio. Tested with "fairness == 0". It seems performance gets better comparing with V6.
Quoted 8 lines
Mode Normal read | Random read | Normal write | Random write | Direct read | Direct Write
2.6.31-rc1 71,613KiB/s 3,606KiB/s 66,250KiB/s 9,420KiB/s 51,535KiB/s 55,752KiB/s
V7 70,540KiB/s 3,551KiB/s 64,548KiB/s 9,677KiB/s 53,530KiB/s 54,145KiB/s
Performance -1.5% -1.5% -2.6% +2.7% +3.9% -2.9%
Thanks Gui. Can you also try V7 with
CONFIG_TRACK_ASYNC_CONTEXT=n. I tried that and I got better results for buffered writes.In my testing I still see some performance regression for buffered writes which goes away if I disable group io scheduling and just use flat mode.
I will spend more time to find out where it is coming from.
Thanks
VivekQuoted 406 lines
Quoted 402 lines · Vivek Goyal
Hi All,
Here is the V7 of the IO controller patches generated on top of 2.6.31-rc4.
For ease of patching, a consolidated patch is available here.
http://people.redhat.com/~vgoyal/io-controller/io-scheduler-based-io-controller-v7.patch
Previous versions of the patches was posted here.
(V1) http://lkml.org/lkml/2009/3/11/486
(V2) http://lkml.org/lkml/2009/5/5/275
(V3) http://lkml.org/lkml/2009/5/26/472
(V4) http://lkml.org/lkml/2009/6/8/580
(V5) http://lkml.org/lkml/2009/6/19/279
(V6) http://lkml.org/lkml/2009/7/2/369Changes from V6
===============
- Introduced the notion ofgroup_idlingwhere we idle for next request to come from the same group before we expire it. It is along the lines of cfq'sslice_idlething to provide fairness. Switching to group idling now helps in the sense that we don't have to rely whether queue idling was turned on or not by CFQ. It becomes too much of debugging pain with different work loads and different kind of storage media. Introduction ofgroup_idleshould help.- Moved some of the code like dynamic queue idling update, arming queue idling timer, keeping track of average think time etc back to CFQ. With group idling we don't need it now. Reduce the amount of change.
- Enabled cfq's close cooperator functionality in groups. So far this worked only in root group. Now it should work in non-root groups also.
- Got rid of the patch where we calculated disk time based on average disk rate in some circumstances. It was giving bad numbers in early queue deletion cases. Also did not think that it was helping a lot. Remvoed it for the time being.
- Added an experimental patch to map sync requests using bio tracking info and not task context. This is only for noop, deadline and AS.
- Got rid of experimental patch of idling for async queues. Don't think it was helping.
- Got rid of
wait_busyandwait_busy_donelogic from queue. Instead implemented it for groups.- Introduced
oom_ioqto accomodateoom_cfqqchange recently.- Broke-up
elv_init_ioq()fn into smaller functions. It had 7 arguments and looked complicated.- Fixed a bug in
blk_queue_io_group_congested(). Thanks to Munehiro Ikeda.- Merged gui's patch to fix the cgroup file format issue.
- Merged gui's patch to update per group congestion limit when
q->nr_group_requestsis updated.- Fixed a bug where close cooperation will not work if we wait for all the requests to finish from previous queue.
- Fixed group deletion accouting where deletion from idle tree were also appearing in the log.
- Got rid of
busy_rt_queuesinfrastructure.- Got rid of
elv_ioq_request_dispatched(). An helper function just to increment a variable.Limitations
===========- This IO controller provides the bandwidth control at the IO scheduler level (leaf node in stacked hiearchy of logical devices). So there can be cases (depending on configuration) where application does not see proportional BW division at higher logical level device.
LWN has written an article about the issue here.
http://lwn.net/Articles/332839/How to solve the issue of fairness at higher level logical devices
==================================================================
(Do we really need it? That's not where the contention for resources is.)Couple of suggestions have come forward.
- Implement IO control at IO scheduler layer and then with the help of some daemon, adjust the weight on underlying devices dynamiclly, depending on what kind of BW gurantees are to be achieved at higher level logical block devices.
- Also implement a higher level IO controller along with IO scheduler based controller and let user choose one depending on his needs.
A higher level controller does not know about the assumptions/policies of unerldying IO scheduler, hence it has the potential to break down the IO scheduler's policy with-in cgroup. A lower level controller can work with IO scheduler much more closely and efficiently.
Other active IO controller developments
=======================================IO throttling
-------------This is a max bandwidth controller and not the proportional one. Secondly it is a second level controller which can break the IO scheduler's policy/assumtions with-in cgroup.
dm-ioband
---------This is a proportional bandwidth controller implemented as device mapper driver. It is also a second level controller which can break the IO scheduler's policy/assumptions with-in cgroup.
TODO
====
- code cleanups, testing, bug fixing, optimizations, benchmarking etc...Testing
=======I have been able to do some testing as follows. All my testing is with ext3 file system with a SATA drive which supports queue depth of 31.
Test1 (Isolation between two KVM virtual machines)
==================================================
Created two KVM virtual machines. Partitioned a disk on host in two partitions and gave one partition to each virtual machine. Put both the virtual machines in two different cgroup of weight 1000 and 500 each. Virtual machines created ext3 file system on the partitions exported from host and did buffered writes. Host seems writes as synchronous and virtual machine with higher weight gets double the disk time of virtual machine of lower weight. Used deadline scheduler in this test case.Some more details about configuration are in documentation patch.
Test2 (Fairness for synchronous reads)
======================================
- Two dd in two cgroups with cgrop weights 1000 and 500. Ran two "dd" in those cgroups (With CFQ scheduler and /sys/block/<device>/queue/fairness = 1)Higher weight dd finishes first and at that point of time my script takes care of reading cgroup files io.disk_time and io.disk_sectors for both the groups and display the results.
dd if=/mnt/$BLOCKDEV/zerofile1 of=/dev/null &
dd if=/mnt/$BLOCKDEV/zerofile2 of=/dev/null &234179072 bytes (234 MB) copied, 3.9065 s, 59.9 MB/s 234179072 bytes (234 MB) copied, 5.19232 s, 45.1 MB/s
group1 time=8 16 2471 group1 sectors=8 16 457840 group2 time=8 16 1220 group2 sectors=8 16 225736
First two fields in time and sectors statistics represent major and minor number of the device. Third field represents disk time in milliseconds and number of sectors transferred respectively.
This patchset tries to provide fairness in terms of disk time received. group1 got almost double of group2 disk time (At the time of first dd finish). These time and sectors statistics can be read using io.disk_time and io.disk_sector files in cgroup. More about it in documentation file.
Test3 (Reader Vs Buffered Writes)
================================
Buffered writes can be problematic and can overwhelm readers, especially with noop and deadline. IO controller can provide isolation between readers and buffered (async) writers.First I ran the test without io controller to see the severity of the issue. Ran a hostile writer and then after 10 seconds started a reader and then monitored the completion time of reader. Reader reads a 256 MB file. Tested this with noop scheduler.
sample script
------------
sync
echo 3 >/proc/sys/vm/drop_caches
time dd if=/dev/zero of=/mnt/sdb/reader-writer-zerofile bs=4K count=2097152 conv=fdatasync &
sleep 10
time dd if=/mnt/sdb/256M-file of=/dev/null &Results
-------
8589934592 bytes (8.6 GB) copied, 106.045 s, 81.0 MB/s (Writer) 268435456 bytes (268 MB) copied, 96.5237 s, 2.8 MB/s (Reader)Now it was time to test io controller whether it can provide isolation between readers and writers with noop. I created two cgroups of weight 1000 each and put reader in group1 and writer in group 2 and ran the test again. Upon comletion of reader, my scripts read io.dis_time and io.disk_group cgroup files to get an estimate how much disk time each group got and how many sectors each group did IO for.
For more accurate accounting of disk time for buffered writes with queuing hardware I had to set /sys/block/<disk>/queue/iosched/fairness to "1".
sample script
-------------
echo $$ > /cgroup/bfqio/test2/tasks
dd if=/dev/zero of=/mnt/$BLOCKDEV/testzerofile bs=4K count=2097152 & sleep 10
echo noop >/sys/block/$BLOCKDEV/queue/scheduler
echo 1 >/sys/block/$BLOCKDEV/queue/iosched/fairnessecho $$ > /cgroup/bfqio/test1/tasks
dd if=/mnt/$BLOCKDEV/256M-file of=/dev/null &
wait $!
# Some code for reading cgroup files upon completion of reader.
-------------------------Results
=======
268435456 bytes (268 MB) copied, 6.65819 s, 40.3 MB/s (Reader)group1 time=8 16 3063 group1 sectors=8 16 524808
group2 time=8 16 3071 group2 sectors=8 16 441752Note, reader finishes now much lesser time and both group1 and group2 got almost 3 seconds of disk time. Hence io-controller provides isolation from buffered writes.
Test4 (AIO)
===========AIO reads
-----------
Set up two fio, AIO read jobs in two cgroup with weight 1000 and 500 respectively. I am using cfq scheduler. Following are some lines from my test script.---------------------------------------------------------------
echo 1000 > /cgroup/bfqio/test1/io.weight
echo 500 > /cgroup/bfqio/test2/io.weightfio_args="--ioengine=libaio --rw=read --size=512M --direct=1" echo 1 >
/sys/block/$BLOCKDEV/queue/iosched/fairnessecho $$ > /cgroup/bfqio/test1/tasks
fio $fio_args --name=test1 --directory=/mnt/$BLOCKDEV/fio1/ --output=/mnt/$BLOCKDEV/fio1/test1.log --exec_postrun="../read-and-display-group-stats.sh $maj_dev $minor_dev" &echo $$ > /cgroup/bfqio/test2/tasks
fio $fio_args --name=test2 --directory=/mnt/$BLOCKDEV/fio2/ --output=/mnt/$BLOCKDEV/fio2/test2.log &
----------------------------------------------------------------test1 and test2 are two groups with weight 1000 and 500 respectively. "
read-and-display-group-stats.sh" is one small script which reads the test1 and test2 cgroup files to determine how much disk time each group got till first fio job finished.Results
------
test1 statistics: time=8 16 22403 sectors=8 16 1049640
test2 statistics: time=8 16 11400 sectors=8 16 552864Above shows that by the time first fio (higher weight), finished, group test1 got 22403 ms of disk time and group test2 got 11400 ms of disk time. similarly the statistics for number of sectors transferred are also shown.
Note that disk time given to group test1 is almost double of group2 disk time.
AIO writes
----------
Set up two fio, AIO direct write jobs in two cgroup with weight 1000 and 500 respectively. I am using cfq scheduler. Following are some lines from my test script.------------------------------------------------
echo 1000 > /cgroup/bfqio/test1/io.weight
echo 500 > /cgroup/bfqio/test2/io.weight fio_args="--ioengine=libaio --rw=write --size=512M --direct=1"echo 1 >
/sys/block/$BLOCKDEV/queue/iosched/fairnessecho $$ > /cgroup/bfqio/test1/tasks
fio $fio_args --name=test1 --directory=/mnt/$BLOCKDEV/fio1/ --output=/mnt/$BLOCKDEV/fio1/test1.log --exec_postrun="../read-and-display-group-stats.sh $maj_dev $minor_dev" &echo $$ > /cgroup/bfqio/test2/tasks
fio $fio_args --name=test2 --directory=/mnt/$BLOCKDEV/fio2/ --output=/mnt/$BLOCKDEV/fio2/test2.log &
-------------------------------------------------test1 and test2 are two groups with weight 1000 and 500 respectively. "
read-and-display-group-stats.sh" is one small script which reads the test1 and test2 cgroup files to determine how much disk time each group got till first fio job finished.Following are the results.
test1 statistics: time=8 16 29085 sectors=8 16 1049656
test2 statistics: time=8 16 14652 sectors=8 16 516728Above shows that by the time first fio (higher weight), finished, group test1 got 28085 ms of disk time and group test2 got 14652 ms of disk time. similarly the statistics for number of sectors transferred are also shown.
Note that disk time given to group test1 is almost double of group2 disk time.
Test5 (Fairness for async writes, Buffered Write Vs Buffered Write)
===================================================================
Fairness for async writes is tricky and biggest reason is that async writes are cached in higher layers (page cahe) as well as possibly in file system layer also (btrfs, xfs etc), and are dispatched to lower layers not necessarily in proportional manner.For example, consider two dd threads reading
/dev/zeroas input file and doing writes of huge files. Very soon we will crossvm_dirty_ratioand dd thread will be forced to write out some pages to disk before more pages can be dirtied. But not necessarily dirty pages of same thread are picked. It can very well pick the inode of lesser priority dd thread and do some writeout. So effectively higher weight dd is doing writeouts of lower weight dd pages and we don't see service differentation.IOW, the core problem with async write fairness is that higher weight thread does not throw enought IO traffic at IO controller to keep the queue continuously backlogged. In my testing, there are many .2 to .8 second intervals where higher weight queue is empty and in that duration lower weight queue get lots of job done giving the impression that there was no service differentiation.
In summary, from IO controller point of view async writes support is there. Because page cache has not been designed in such a manner that higher prio/weight writer can do more write out as compared to lower prio/weight writer, gettting service differentiation is hard and it is visible in some cases and not visible in some cases.
Do we really care that much for fairness among two writer cgroups? One can choose to do direct writes or sync writes if fairness for writes really matters for him.
Following is the only case where it is hard to ensure fairness between cgroups.
- Buffered writes Vs Buffered Writes.
So to test async writes I created two partitions on a disk and created ext3 file systems on both the partitions. Also created two cgroups and generated lots of write traffic in two cgroups (50 fio threads) and watched the disk time statistics in respective cgroups at the interval of 2 seconds. Thanks to ryo tsuruta for the test case.
*****************************************************************
sync
echo 3 >/proc/sys/vm/drop_cachesfio_args="--size=64m --rw=write --numjobs=50 --group_reporting"
echo $$ > /cgroup/bfqio/test1/tasks
fio $fio_args --name=test1 --directory=/mnt/sdd1/fio/ --output=/mnt/sdd1/fio/test1.log &echo $$ > /cgroup/bfqio/test2/tasks
fio $fio_args --name=test2 --directory=/mnt/sdd2/fio/ --output=/mnt/sdd2/fio/test2.log &
***********************************************************************And watched the disk time and sector statistics for the both the cgroups every 2 seconds using a script. How is snippet from output.
test1 statistics: time=8 48 1315 sectors=8 48 55776 dq=8 48 1
test2 statistics: time=8 48 633 sectors=8 48 14720 dq=8 48 2test1 statistics: time=8 48 5586 sectors=8 48 339064 dq=8 48 2
test2 statistics: time=8 48 2985 sectors=8 48 146656 dq=8 48 3test1 statistics: time=8 48 9935 sectors=8 48 628728 dq=8 48 3
test2 statistics: time=8 48 5265 sectors=8 48 278688 dq=8 48 4test1 statistics: time=8 48 14156 sectors=8 48 932488 dq=8 48 6
test2 statistics: time=8 48 7646 sectors=8 48 412704 dq=8 48 7test1 statistics: time=8 48 18141 sectors=8 48 1231488 dq=8 48 10
test2 statistics: time=8 48 9820 sectors=8 48 548400 dq=8 48 8test1 statistics: time=8 48 21953 sectors=8 48 1485632 dq=8 48 13
test2 statistics: time=8 48 12394 sectors=8 48 698288 dq=8 48 10test1 statistics: time=8 48 25167 sectors=8 48 1705264 dq=8 48 13
test2 statistics: time=8 48 14042 sectors=8 48 817808 dq=8 48 10First two fields in time and sectors statistics represent major and minor number of the device. Third field represents disk time in milliseconds and number of sectors transferred respectively.
So disk time consumed by group1 is almost double of group2 in this case.
Your feedback is welcome.
Thanks
VivekSignature · 2 lines
Regards Gui Jianfeng
Signature · 2 lines
Regards Gui Jianfeng
Quoted 11 lines · Gui Jianfeng
Vivek, Here are some test results with and without
CONFIG_TRACK_ASYNC_CONTEXTfor V7Mode Normal read | Random read | Normal write | Random write | Direct read | Direct Write
CONFIG_TRACK_ASYNC_CONTEXT=y70,540KiB/s 3,551KiB/s 64,548KiB/s 9,677KiB/s 53,530KiB/s 54,145KiB/s
CONFIG_TRACK_ASYNC_CONTEXT=n71,082KiB/s 3,564KiB/s 66,720KiB/s 9,887KiB/s 51,401KiB/s 55,210KiB/sPerformance +0.7% +0.3% +3.3% +2.1% -4.0% +2.0%
Strange. Disabling async context tracking should not impact read performance as reads are always sync and don't take async tracking path even if it is enabled. We are instead seeing -4% in direct reads if track async context is disabled.
I would recommend that there can be lot of variance between multiple runs. We should probably run each test 3 times and take some average of that.
Thanks
Vivek
Quoted 442 lines
Quoted 436 lines · Vivek Goyal
Quoted 5 lines · Gui Jianfeng
Hi Vivek,
Here are some test results for normal reads and write for IO Controller V7 by fio. Tested with "fairness == 0". It seems performance gets better comparing with V6.
Quoted 8 lines
Mode Normal read | Random read | Normal write | Random write | Direct read | Direct Write
2.6.31-rc1 71,613KiB/s 3,606KiB/s 66,250KiB/s 9,420KiB/s 51,535KiB/s 55,752KiB/s
V7 70,540KiB/s 3,551KiB/s 64,548KiB/s 9,677KiB/s 53,530KiB/s 54,145KiB/s
Performance -1.5% -1.5% -2.6% +2.7% +3.9% -2.9%
Thanks Gui. Can you also try V7 with
CONFIG_TRACK_ASYNC_CONTEXT=n. I tried that and I got better results for buffered writes.In my testing I still see some performance regression for buffered writes which goes away if I disable group io scheduling and just use flat mode.
I will spend more time to find out where it is coming from.
Thanks
VivekQuoted 406 lines
Quoted 402 lines · Vivek Goyal
Hi All,
Here is the V7 of the IO controller patches generated on top of 2.6.31-rc4.
For ease of patching, a consolidated patch is available here.
http://people.redhat.com/~vgoyal/io-controller/io-scheduler-based-io-controller-v7.patch
Previous versions of the patches was posted here.
(V1) http://lkml.org/lkml/2009/3/11/486
(V2) http://lkml.org/lkml/2009/5/5/275
(V3) http://lkml.org/lkml/2009/5/26/472
(V4) http://lkml.org/lkml/2009/6/8/580
(V5) http://lkml.org/lkml/2009/6/19/279
(V6) http://lkml.org/lkml/2009/7/2/369Changes from V6
===============
- Introduced the notion ofgroup_idlingwhere we idle for next request to come from the same group before we expire it. It is along the lines of cfq'sslice_idlething to provide fairness. Switching to group idling now helps in the sense that we don't have to rely whether queue idling was turned on or not by CFQ. It becomes too much of debugging pain with different work loads and different kind of storage media. Introduction ofgroup_idleshould help.- Moved some of the code like dynamic queue idling update, arming queue idling timer, keeping track of average think time etc back to CFQ. With group idling we don't need it now. Reduce the amount of change.
- Enabled cfq's close cooperator functionality in groups. So far this worked only in root group. Now it should work in non-root groups also.
- Got rid of the patch where we calculated disk time based on average disk rate in some circumstances. It was giving bad numbers in early queue deletion cases. Also did not think that it was helping a lot. Remvoed it for the time being.
- Added an experimental patch to map sync requests using bio tracking info and not task context. This is only for noop, deadline and AS.
- Got rid of experimental patch of idling for async queues. Don't think it was helping.
- Got rid of
wait_busyandwait_busy_donelogic from queue. Instead implemented it for groups.- Introduced
oom_ioqto accomodateoom_cfqqchange recently.- Broke-up
elv_init_ioq()fn into smaller functions. It had 7 arguments and looked complicated.- Fixed a bug in
blk_queue_io_group_congested(). Thanks to Munehiro Ikeda.- Merged gui's patch to fix the cgroup file format issue.
- Merged gui's patch to update per group congestion limit when
q->nr_group_requestsis updated.- Fixed a bug where close cooperation will not work if we wait for all the requests to finish from previous queue.
- Fixed group deletion accouting where deletion from idle tree were also appearing in the log.
- Got rid of
busy_rt_queuesinfrastructure.- Got rid of
elv_ioq_request_dispatched(). An helper function just to increment a variable.Limitations
===========- This IO controller provides the bandwidth control at the IO scheduler level (leaf node in stacked hiearchy of logical devices). So there can be cases (depending on configuration) where application does not see proportional BW division at higher logical level device.
LWN has written an article about the issue here.
http://lwn.net/Articles/332839/How to solve the issue of fairness at higher level logical devices
==================================================================
(Do we really need it? That's not where the contention for resources is.)Couple of suggestions have come forward.
- Implement IO control at IO scheduler layer and then with the help of some daemon, adjust the weight on underlying devices dynamiclly, depending on what kind of BW gurantees are to be achieved at higher level logical block devices.
- Also implement a higher level IO controller along with IO scheduler based controller and let user choose one depending on his needs.
A higher level controller does not know about the assumptions/policies of unerldying IO scheduler, hence it has the potential to break down the IO scheduler's policy with-in cgroup. A lower level controller can work with IO scheduler much more closely and efficiently.
Other active IO controller developments
=======================================IO throttling
-------------This is a max bandwidth controller and not the proportional one. Secondly it is a second level controller which can break the IO scheduler's policy/assumtions with-in cgroup.
dm-ioband
---------This is a proportional bandwidth controller implemented as device mapper driver. It is also a second level controller which can break the IO scheduler's policy/assumptions with-in cgroup.
TODO
====
- code cleanups, testing, bug fixing, optimizations, benchmarking etc...Testing
=======I have been able to do some testing as follows. All my testing is with ext3 file system with a SATA drive which supports queue depth of 31.
Test1 (Isolation between two KVM virtual machines)
==================================================
Created two KVM virtual machines. Partitioned a disk on host in two partitions and gave one partition to each virtual machine. Put both the virtual machines in two different cgroup of weight 1000 and 500 each. Virtual machines created ext3 file system on the partitions exported from host and did buffered writes. Host seems writes as synchronous and virtual machine with higher weight gets double the disk time of virtual machine of lower weight. Used deadline scheduler in this test case.Some more details about configuration are in documentation patch.
Test2 (Fairness for synchronous reads)
======================================
- Two dd in two cgroups with cgrop weights 1000 and 500. Ran two "dd" in those cgroups (With CFQ scheduler and /sys/block/<device>/queue/fairness = 1)Higher weight dd finishes first and at that point of time my script takes care of reading cgroup files io.disk_time and io.disk_sectors for both the groups and display the results.
dd if=/mnt/$BLOCKDEV/zerofile1 of=/dev/null &
dd if=/mnt/$BLOCKDEV/zerofile2 of=/dev/null &234179072 bytes (234 MB) copied, 3.9065 s, 59.9 MB/s 234179072 bytes (234 MB) copied, 5.19232 s, 45.1 MB/s
group1 time=8 16 2471 group1 sectors=8 16 457840 group2 time=8 16 1220 group2 sectors=8 16 225736
First two fields in time and sectors statistics represent major and minor number of the device. Third field represents disk time in milliseconds and number of sectors transferred respectively.
This patchset tries to provide fairness in terms of disk time received. group1 got almost double of group2 disk time (At the time of first dd finish). These time and sectors statistics can be read using io.disk_time and io.disk_sector files in cgroup. More about it in documentation file.
Test3 (Reader Vs Buffered Writes)
================================
Buffered writes can be problematic and can overwhelm readers, especially with noop and deadline. IO controller can provide isolation between readers and buffered (async) writers.First I ran the test without io controller to see the severity of the issue. Ran a hostile writer and then after 10 seconds started a reader and then monitored the completion time of reader. Reader reads a 256 MB file. Tested this with noop scheduler.
sample script
------------
sync
echo 3 >/proc/sys/vm/drop_caches
time dd if=/dev/zero of=/mnt/sdb/reader-writer-zerofile bs=4K count=2097152 conv=fdatasync &
sleep 10
time dd if=/mnt/sdb/256M-file of=/dev/null &Results
-------
8589934592 bytes (8.6 GB) copied, 106.045 s, 81.0 MB/s (Writer) 268435456 bytes (268 MB) copied, 96.5237 s, 2.8 MB/s (Reader)Now it was time to test io controller whether it can provide isolation between readers and writers with noop. I created two cgroups of weight 1000 each and put reader in group1 and writer in group 2 and ran the test again. Upon comletion of reader, my scripts read io.dis_time and io.disk_group cgroup files to get an estimate how much disk time each group got and how many sectors each group did IO for.
For more accurate accounting of disk time for buffered writes with queuing hardware I had to set /sys/block/<disk>/queue/iosched/fairness to "1".
sample script
-------------
echo $$ > /cgroup/bfqio/test2/tasks
dd if=/dev/zero of=/mnt/$BLOCKDEV/testzerofile bs=4K count=2097152 & sleep 10
echo noop >/sys/block/$BLOCKDEV/queue/scheduler
echo 1 >/sys/block/$BLOCKDEV/queue/iosched/fairnessecho $$ > /cgroup/bfqio/test1/tasks
dd if=/mnt/$BLOCKDEV/256M-file of=/dev/null &
wait $!
# Some code for reading cgroup files upon completion of reader.
-------------------------Results
=======
268435456 bytes (268 MB) copied, 6.65819 s, 40.3 MB/s (Reader)group1 time=8 16 3063 group1 sectors=8 16 524808
group2 time=8 16 3071 group2 sectors=8 16 441752Note, reader finishes now much lesser time and both group1 and group2 got almost 3 seconds of disk time. Hence io-controller provides isolation from buffered writes.
Test4 (AIO)
===========AIO reads
-----------
Set up two fio, AIO read jobs in two cgroup with weight 1000 and 500 respectively. I am using cfq scheduler. Following are some lines from my test script.---------------------------------------------------------------
echo 1000 > /cgroup/bfqio/test1/io.weight
echo 500 > /cgroup/bfqio/test2/io.weightfio_args="--ioengine=libaio --rw=read --size=512M --direct=1" echo 1 >
/sys/block/$BLOCKDEV/queue/iosched/fairnessecho $$ > /cgroup/bfqio/test1/tasks
fio $fio_args --name=test1 --directory=/mnt/$BLOCKDEV/fio1/ --output=/mnt/$BLOCKDEV/fio1/test1.log --exec_postrun="../read-and-display-group-stats.sh $maj_dev $minor_dev" &echo $$ > /cgroup/bfqio/test2/tasks
fio $fio_args --name=test2 --directory=/mnt/$BLOCKDEV/fio2/ --output=/mnt/$BLOCKDEV/fio2/test2.log &
----------------------------------------------------------------test1 and test2 are two groups with weight 1000 and 500 respectively. "
read-and-display-group-stats.sh" is one small script which reads the test1 and test2 cgroup files to determine how much disk time each group got till first fio job finished.Results
------
test1 statistics: time=8 16 22403 sectors=8 16 1049640
test2 statistics: time=8 16 11400 sectors=8 16 552864Above shows that by the time first fio (higher weight), finished, group test1 got 22403 ms of disk time and group test2 got 11400 ms of disk time. similarly the statistics for number of sectors transferred are also shown.
Note that disk time given to group test1 is almost double of group2 disk time.
AIO writes
----------
Set up two fio, AIO direct write jobs in two cgroup with weight 1000 and 500 respectively. I am using cfq scheduler. Following are some lines from my test script.------------------------------------------------
echo 1000 > /cgroup/bfqio/test1/io.weight
echo 500 > /cgroup/bfqio/test2/io.weight fio_args="--ioengine=libaio --rw=write --size=512M --direct=1"echo 1 >
/sys/block/$BLOCKDEV/queue/iosched/fairnessecho $$ > /cgroup/bfqio/test1/tasks
fio $fio_args --name=test1 --directory=/mnt/$BLOCKDEV/fio1/ --output=/mnt/$BLOCKDEV/fio1/test1.log --exec_postrun="../read-and-display-group-stats.sh $maj_dev $minor_dev" &echo $$ > /cgroup/bfqio/test2/tasks
fio $fio_args --name=test2 --directory=/mnt/$BLOCKDEV/fio2/ --output=/mnt/$BLOCKDEV/fio2/test2.log &
-------------------------------------------------test1 and test2 are two groups with weight 1000 and 500 respectively. "
read-and-display-group-stats.sh" is one small script which reads the test1 and test2 cgroup files to determine how much disk time each group got till first fio job finished.Following are the results.
test1 statistics: time=8 16 29085 sectors=8 16 1049656
test2 statistics: time=8 16 14652 sectors=8 16 516728Above shows that by the time first fio (higher weight), finished, group test1 got 28085 ms of disk time and group test2 got 14652 ms of disk time. similarly the statistics for number of sectors transferred are also shown.
Note that disk time given to group test1 is almost double of group2 disk time.
Test5 (Fairness for async writes, Buffered Write Vs Buffered Write)
===================================================================
Fairness for async writes is tricky and biggest reason is that async writes are cached in higher layers (page cahe) as well as possibly in file system layer also (btrfs, xfs etc), and are dispatched to lower layers not necessarily in proportional manner.For example, consider two dd threads reading
/dev/zeroas input file and doing writes of huge files. Very soon we will crossvm_dirty_ratioand dd thread will be forced to write out some pages to disk before more pages can be dirtied. But not necessarily dirty pages of same thread are picked. It can very well pick the inode of lesser priority dd thread and do some writeout. So effectively higher weight dd is doing writeouts of lower weight dd pages and we don't see service differentation.IOW, the core problem with async write fairness is that higher weight thread does not throw enought IO traffic at IO controller to keep the queue continuously backlogged. In my testing, there are many .2 to .8 second intervals where higher weight queue is empty and in that duration lower weight queue get lots of job done giving the impression that there was no service differentiation.
In summary, from IO controller point of view async writes support is there. Because page cache has not been designed in such a manner that higher prio/weight writer can do more write out as compared to lower prio/weight writer, gettting service differentiation is hard and it is visible in some cases and not visible in some cases.
Do we really care that much for fairness among two writer cgroups? One can choose to do direct writes or sync writes if fairness for writes really matters for him.
Following is the only case where it is hard to ensure fairness between cgroups.
- Buffered writes Vs Buffered Writes.
So to test async writes I created two partitions on a disk and created ext3 file systems on both the partitions. Also created two cgroups and generated lots of write traffic in two cgroups (50 fio threads) and watched the disk time statistics in respective cgroups at the interval of 2 seconds. Thanks to ryo tsuruta for the test case.
*****************************************************************
sync
echo 3 >/proc/sys/vm/drop_cachesfio_args="--size=64m --rw=write --numjobs=50 --group_reporting"
echo $$ > /cgroup/bfqio/test1/tasks
fio $fio_args --name=test1 --directory=/mnt/sdd1/fio/ --output=/mnt/sdd1/fio/test1.log &echo $$ > /cgroup/bfqio/test2/tasks
fio $fio_args --name=test2 --directory=/mnt/sdd2/fio/ --output=/mnt/sdd2/fio/test2.log &
***********************************************************************And watched the disk time and sector statistics for the both the cgroups every 2 seconds using a script. How is snippet from output.
test1 statistics: time=8 48 1315 sectors=8 48 55776 dq=8 48 1
test2 statistics: time=8 48 633 sectors=8 48 14720 dq=8 48 2test1 statistics: time=8 48 5586 sectors=8 48 339064 dq=8 48 2
test2 statistics: time=8 48 2985 sectors=8 48 146656 dq=8 48 3test1 statistics: time=8 48 9935 sectors=8 48 628728 dq=8 48 3
test2 statistics: time=8 48 5265 sectors=8 48 278688 dq=8 48 4test1 statistics: time=8 48 14156 sectors=8 48 932488 dq=8 48 6
test2 statistics: time=8 48 7646 sectors=8 48 412704 dq=8 48 7test1 statistics: time=8 48 18141 sectors=8 48 1231488 dq=8 48 10
test2 statistics: time=8 48 9820 sectors=8 48 548400 dq=8 48 8test1 statistics: time=8 48 21953 sectors=8 48 1485632 dq=8 48 13
test2 statistics: time=8 48 12394 sectors=8 48 698288 dq=8 48 10test1 statistics: time=8 48 25167 sectors=8 48 1705264 dq=8 48 13
test2 statistics: time=8 48 14042 sectors=8 48 817808 dq=8 48 10First two fields in time and sectors statistics represent major and minor number of the device. Third field represents disk time in milliseconds and number of sectors transferred respectively.
So disk time consumed by group1 is almost double of group2 in this case.
Your feedback is welcome.
Thanks
VivekSignature · 2 lines
Regards Gui JianfengSignature · 2 lines
Regards Gui Jianfeng
Quoted 13 lines · Vivek Goyal
Quoted 11 lines · Gui Jianfeng
Vivek, Here are some test results with and without
CONFIG_TRACK_ASYNC_CONTEXTfor V7Mode Normal read | Random read | Normal write | Random write | Direct read | Direct Write
CONFIG_TRACK_ASYNC_CONTEXT=y70,540KiB/s 3,551KiB/s 64,548KiB/s 9,677KiB/s 53,530KiB/s 54,145KiB/s
CONFIG_TRACK_ASYNC_CONTEXT=n71,082KiB/s 3,564KiB/s 66,720KiB/s 9,887KiB/s 51,401KiB/s 55,210KiB/sPerformance +0.7% +0.3% +3.3% +2.1% -4.0% +2.0%
Quoted 7 lines
Strange. Disabling async context tracking should not impact read performance as reads are always sync and don't take async tracking path even if it is enabled. We are instead seeing -4% in direct reads if track async context is disabled.
I would recommend that there can be lot of variance between multiple runs. We should probably run each test 3 times and take some average of that.
Sorry for the late reply.
I tried to test direct reads for 5 times when CONFIG_TRACK_ASYNC_CONTEXT=n and y. I got the following results, and still had the performance variance.
For V7.
1st 2nd 3rd 4th 5th avg
CONFIG_TRACK_ASYNC_CONTEXT=y 58,391KiB/s 58,861KiB/s 58,685KiB/s 59,020KiB/s 58,883KiB/s 58,786KiB/s
CONFIG_TRACK_ASYNC_CONTEXT=n 57,045KiB/s 57,827KiB/s 57,744KiB/s 56,884KiB/s 57,821KiB/s 57,619KiB/s
Performance -2.3% -1.7% -1.6% -3.6% -1.8% -2.0%
Quoted 446 lines
Thanks
VivekQuoted 439 lines
Quoted 434 lines · Vivek Goyal
Quoted 5 lines · Gui Jianfeng
Hi Vivek,
Here are some test results for normal reads and write for IO Controller V7 by fio. Tested with "fairness == 0". It seems performance gets better comparing with V6.
Quoted 8 lines
Mode Normal read | Random read | Normal write | Random write | Direct read | Direct Write
2.6.31-rc1 71,613KiB/s 3,606KiB/s 66,250KiB/s 9,420KiB/s 51,535KiB/s 55,752KiB/s
V7 70,540KiB/s 3,551KiB/s 64,548KiB/s 9,677KiB/s 53,530KiB/s 54,145KiB/s
Performance -1.5% -1.5% -2.6% +2.7% +3.9% -2.9%
Thanks Gui. Can you also try V7 with
CONFIG_TRACK_ASYNC_CONTEXT=n. I tried that and I got better results for buffered writes.In my testing I still see some performance regression for buffered writes which goes away if I disable group io scheduling and just use flat mode.
I will spend more time to find out where it is coming from.
Thanks
VivekQuoted 406 lines
Quoted 402 lines · Vivek Goyal
Hi All,
Here is the V7 of the IO controller patches generated on top of 2.6.31-rc4.
For ease of patching, a consolidated patch is available here.
http://people.redhat.com/~vgoyal/io-controller/io-scheduler-based-io-controller-v7.patch
Previous versions of the patches was posted here.
(V1) http://lkml.org/lkml/2009/3/11/486
(V2) http://lkml.org/lkml/2009/5/5/275
(V3) http://lkml.org/lkml/2009/5/26/472
(V4) http://lkml.org/lkml/2009/6/8/580
(V5) http://lkml.org/lkml/2009/6/19/279
(V6) http://lkml.org/lkml/2009/7/2/369Changes from V6
===============
- Introduced the notion ofgroup_idlingwhere we idle for next request to come from the same group before we expire it. It is along the lines of cfq'sslice_idlething to provide fairness. Switching to group idling now helps in the sense that we don't have to rely whether queue idling was turned on or not by CFQ. It becomes too much of debugging pain with different work loads and different kind of storage media. Introduction ofgroup_idleshould help.- Moved some of the code like dynamic queue idling update, arming queue idling timer, keeping track of average think time etc back to CFQ. With group idling we don't need it now. Reduce the amount of change.
- Enabled cfq's close cooperator functionality in groups. So far this worked only in root group. Now it should work in non-root groups also.
- Got rid of the patch where we calculated disk time based on average disk rate in some circumstances. It was giving bad numbers in early queue deletion cases. Also did not think that it was helping a lot. Remvoed it for the time being.
- Added an experimental patch to map sync requests using bio tracking info and not task context. This is only for noop, deadline and AS.
- Got rid of experimental patch of idling for async queues. Don't think it was helping.
- Got rid of
wait_busyandwait_busy_donelogic from queue. Instead implemented it for groups.- Introduced
oom_ioqto accomodateoom_cfqqchange recently.- Broke-up
elv_init_ioq()fn into smaller functions. It had 7 arguments and looked complicated.- Fixed a bug in
blk_queue_io_group_congested(). Thanks to Munehiro Ikeda.- Merged gui's patch to fix the cgroup file format issue.
- Merged gui's patch to update per group congestion limit when
q->nr_group_requestsis updated.- Fixed a bug where close cooperation will not work if we wait for all the requests to finish from previous queue.
- Fixed group deletion accouting where deletion from idle tree were also appearing in the log.
- Got rid of
busy_rt_queuesinfrastructure.- Got rid of
elv_ioq_request_dispatched(). An helper function just to increment a variable.Limitations
===========- This IO controller provides the bandwidth control at the IO scheduler level (leaf node in stacked hiearchy of logical devices). So there can be cases (depending on configuration) where application does not see proportional BW division at higher logical level device.
LWN has written an article about the issue here.
http://lwn.net/Articles/332839/How to solve the issue of fairness at higher level logical devices
==================================================================
(Do we really need it? That's not where the contention for resources is.)Couple of suggestions have come forward.
- Implement IO control at IO scheduler layer and then with the help of some daemon, adjust the weight on underlying devices dynamiclly, depending on what kind of BW gurantees are to be achieved at higher level logical block devices.
- Also implement a higher level IO controller along with IO scheduler based controller and let user choose one depending on his needs.
A higher level controller does not know about the assumptions/policies of unerldying IO scheduler, hence it has the potential to break down the IO scheduler's policy with-in cgroup. A lower level controller can work with IO scheduler much more closely and efficiently.
Other active IO controller developments
=======================================IO throttling
-------------This is a max bandwidth controller and not the proportional one. Secondly it is a second level controller which can break the IO scheduler's policy/assumtions with-in cgroup.
dm-ioband
---------This is a proportional bandwidth controller implemented as device mapper driver. It is also a second level controller which can break the IO scheduler's policy/assumptions with-in cgroup.
TODO
====
- code cleanups, testing, bug fixing, optimizations, benchmarking etc...Testing
=======I have been able to do some testing as follows. All my testing is with ext3 file system with a SATA drive which supports queue depth of 31.
Test1 (Isolation between two KVM virtual machines)
==================================================
Created two KVM virtual machines. Partitioned a disk on host in two partitions and gave one partition to each virtual machine. Put both the virtual machines in two different cgroup of weight 1000 and 500 each. Virtual machines created ext3 file system on the partitions exported from host and did buffered writes. Host seems writes as synchronous and virtual machine with higher weight gets double the disk time of virtual machine of lower weight. Used deadline scheduler in this test case.Some more details about configuration are in documentation patch.
Test2 (Fairness for synchronous reads)
======================================
- Two dd in two cgroups with cgrop weights 1000 and 500. Ran two "dd" in those cgroups (With CFQ scheduler and /sys/block/<device>/queue/fairness = 1)Higher weight dd finishes first and at that point of time my script takes care of reading cgroup files io.disk_time and io.disk_sectors for both the groups and display the results.
dd if=/mnt/$BLOCKDEV/zerofile1 of=/dev/null &
dd if=/mnt/$BLOCKDEV/zerofile2 of=/dev/null &234179072 bytes (234 MB) copied, 3.9065 s, 59.9 MB/s 234179072 bytes (234 MB) copied, 5.19232 s, 45.1 MB/s
group1 time=8 16 2471 group1 sectors=8 16 457840 group2 time=8 16 1220 group2 sectors=8 16 225736
First two fields in time and sectors statistics represent major and minor number of the device. Third field represents disk time in milliseconds and number of sectors transferred respectively.
This patchset tries to provide fairness in terms of disk time received. group1 got almost double of group2 disk time (At the time of first dd finish). These time and sectors statistics can be read using io.disk_time and io.disk_sector files in cgroup. More about it in documentation file.
Test3 (Reader Vs Buffered Writes)
================================
Buffered writes can be problematic and can overwhelm readers, especially with noop and deadline. IO controller can provide isolation between readers and buffered (async) writers.First I ran the test without io controller to see the severity of the issue. Ran a hostile writer and then after 10 seconds started a reader and then monitored the completion time of reader. Reader reads a 256 MB file. Tested this with noop scheduler.
sample script
------------
sync
echo 3 >/proc/sys/vm/drop_caches
time dd if=/dev/zero of=/mnt/sdb/reader-writer-zerofile bs=4K count=2097152 conv=fdatasync &
sleep 10
time dd if=/mnt/sdb/256M-file of=/dev/null &Results
-------
8589934592 bytes (8.6 GB) copied, 106.045 s, 81.0 MB/s (Writer) 268435456 bytes (268 MB) copied, 96.5237 s, 2.8 MB/s (Reader)Now it was time to test io controller whether it can provide isolation between readers and writers with noop. I created two cgroups of weight 1000 each and put reader in group1 and writer in group 2 and ran the test again. Upon comletion of reader, my scripts read io.dis_time and io.disk_group cgroup files to get an estimate how much disk time each group got and how many sectors each group did IO for.
For more accurate accounting of disk time for buffered writes with queuing hardware I had to set /sys/block/<disk>/queue/iosched/fairness to "1".
sample script
-------------
echo $$ > /cgroup/bfqio/test2/tasks
dd if=/dev/zero of=/mnt/$BLOCKDEV/testzerofile bs=4K count=2097152 & sleep 10
echo noop >/sys/block/$BLOCKDEV/queue/scheduler
echo 1 >/sys/block/$BLOCKDEV/queue/iosched/fairnessecho $$ > /cgroup/bfqio/test1/tasks
dd if=/mnt/$BLOCKDEV/256M-file of=/dev/null &
wait $!
# Some code for reading cgroup files upon completion of reader.
-------------------------Results
=======
268435456 bytes (268 MB) copied, 6.65819 s, 40.3 MB/s (Reader)group1 time=8 16 3063 group1 sectors=8 16 524808
group2 time=8 16 3071 group2 sectors=8 16 441752Note, reader finishes now much lesser time and both group1 and group2 got almost 3 seconds of disk time. Hence io-controller provides isolation from buffered writes.
Test4 (AIO)
===========AIO reads
-----------
Set up two fio, AIO read jobs in two cgroup with weight 1000 and 500 respectively. I am using cfq scheduler. Following are some lines from my test script.---------------------------------------------------------------
echo 1000 > /cgroup/bfqio/test1/io.weight
echo 500 > /cgroup/bfqio/test2/io.weightfio_args="--ioengine=libaio --rw=read --size=512M --direct=1" echo 1 >
/sys/block/$BLOCKDEV/queue/iosched/fairnessecho $$ > /cgroup/bfqio/test1/tasks
fio $fio_args --name=test1 --directory=/mnt/$BLOCKDEV/fio1/ --output=/mnt/$BLOCKDEV/fio1/test1.log --exec_postrun="../read-and-display-group-stats.sh $maj_dev $minor_dev" &echo $$ > /cgroup/bfqio/test2/tasks
fio $fio_args --name=test2 --directory=/mnt/$BLOCKDEV/fio2/ --output=/mnt/$BLOCKDEV/fio2/test2.log &
----------------------------------------------------------------test1 and test2 are two groups with weight 1000 and 500 respectively. "
read-and-display-group-stats.sh" is one small script which reads the test1 and test2 cgroup files to determine how much disk time each group got till first fio job finished.Results
------
test1 statistics: time=8 16 22403 sectors=8 16 1049640
test2 statistics: time=8 16 11400 sectors=8 16 552864Above shows that by the time first fio (higher weight), finished, group test1 got 22403 ms of disk time and group test2 got 11400 ms of disk time. similarly the statistics for number of sectors transferred are also shown.
Note that disk time given to group test1 is almost double of group2 disk time.
AIO writes
----------
Set up two fio, AIO direct write jobs in two cgroup with weight 1000 and 500 respectively. I am using cfq scheduler. Following are some lines from my test script.------------------------------------------------
echo 1000 > /cgroup/bfqio/test1/io.weight
echo 500 > /cgroup/bfqio/test2/io.weight fio_args="--ioengine=libaio --rw=write --size=512M --direct=1"echo 1 >
/sys/block/$BLOCKDEV/queue/iosched/fairnessecho $$ > /cgroup/bfqio/test1/tasks
fio $fio_args --name=test1 --directory=/mnt/$BLOCKDEV/fio1/ --output=/mnt/$BLOCKDEV/fio1/test1.log --exec_postrun="../read-and-display-group-stats.sh $maj_dev $minor_dev" &echo $$ > /cgroup/bfqio/test2/tasks
fio $fio_args --name=test2 --directory=/mnt/$BLOCKDEV/fio2/ --output=/mnt/$BLOCKDEV/fio2/test2.log &
-------------------------------------------------test1 and test2 are two groups with weight 1000 and 500 respectively. "
read-and-display-group-stats.sh" is one small script which reads the test1 and test2 cgroup files to determine how much disk time each group got till first fio job finished.Following are the results.
test1 statistics: time=8 16 29085 sectors=8 16 1049656
test2 statistics: time=8 16 14652 sectors=8 16 516728Above shows that by the time first fio (higher weight), finished, group test1 got 28085 ms of disk time and group test2 got 14652 ms of disk time. similarly the statistics for number of sectors transferred are also shown.
Note that disk time given to group test1 is almost double of group2 disk time.
Test5 (Fairness for async writes, Buffered Write Vs Buffered Write)
===================================================================
Fairness for async writes is tricky and biggest reason is that async writes are cached in higher layers (page cahe) as well as possibly in file system layer also (btrfs, xfs etc), and are dispatched to lower layers not necessarily in proportional manner.For example, consider two dd threads reading
/dev/zeroas input file and doing writes of huge files. Very soon we will crossvm_dirty_ratioand dd thread will be forced to write out some pages to disk before more pages can be dirtied. But not necessarily dirty pages of same thread are picked. It can very well pick the inode of lesser priority dd thread and do some writeout. So effectively higher weight dd is doing writeouts of lower weight dd pages and we don't see service differentation.IOW, the core problem with async write fairness is that higher weight thread does not throw enought IO traffic at IO controller to keep the queue continuously backlogged. In my testing, there are many .2 to .8 second intervals where higher weight queue is empty and in that duration lower weight queue get lots of job done giving the impression that there was no service differentiation.
In summary, from IO controller point of view async writes support is there. Because page cache has not been designed in such a manner that higher prio/weight writer can do more write out as compared to lower prio/weight writer, gettting service differentiation is hard and it is visible in some cases and not visible in some cases.
Do we really care that much for fairness among two writer cgroups? One can choose to do direct writes or sync writes if fairness for writes really matters for him.
Following is the only case where it is hard to ensure fairness between cgroups.
- Buffered writes Vs Buffered Writes.
So to test async writes I created two partitions on a disk and created ext3 file systems on both the partitions. Also created two cgroups and generated lots of write traffic in two cgroups (50 fio threads) and watched the disk time statistics in respective cgroups at the interval of 2 seconds. Thanks to ryo tsuruta for the test case.
*****************************************************************
sync
echo 3 >/proc/sys/vm/drop_cachesfio_args="--size=64m --rw=write --numjobs=50 --group_reporting"
echo $$ > /cgroup/bfqio/test1/tasks
fio $fio_args --name=test1 --directory=/mnt/sdd1/fio/ --output=/mnt/sdd1/fio/test1.log &echo $$ > /cgroup/bfqio/test2/tasks
fio $fio_args --name=test2 --directory=/mnt/sdd2/fio/ --output=/mnt/sdd2/fio/test2.log &
***********************************************************************And watched the disk time and sector statistics for the both the cgroups every 2 seconds using a script. How is snippet from output.
test1 statistics: time=8 48 1315 sectors=8 48 55776 dq=8 48 1
test2 statistics: time=8 48 633 sectors=8 48 14720 dq=8 48 2test1 statistics: time=8 48 5586 sectors=8 48 339064 dq=8 48 2
test2 statistics: time=8 48 2985 sectors=8 48 146656 dq=8 48 3test1 statistics: time=8 48 9935 sectors=8 48 628728 dq=8 48 3
test2 statistics: time=8 48 5265 sectors=8 48 278688 dq=8 48 4test1 statistics: time=8 48 14156 sectors=8 48 932488 dq=8 48 6
test2 statistics: time=8 48 7646 sectors=8 48 412704 dq=8 48 7test1 statistics: time=8 48 18141 sectors=8 48 1231488 dq=8 48 10
test2 statistics: time=8 48 9820 sectors=8 48 548400 dq=8 48 8test1 statistics: time=8 48 21953 sectors=8 48 1485632 dq=8 48 13
test2 statistics: time=8 48 12394 sectors=8 48 698288 dq=8 48 10test1 statistics: time=8 48 25167 sectors=8 48 1705264 dq=8 48 13
test2 statistics: time=8 48 14042 sectors=8 48 817808 dq=8 48 10First two fields in time and sectors statistics represent major and minor number of the device. Third field represents disk time in milliseconds and number of sectors transferred respectively.
So disk time consumed by group1 is almost double of group2 in this case.
Your feedback is welcome.
Thanks
VivekSignature · 2 lines
Regards Gui JianfengSignature · 2 lines
Regards Gui Jianfeng
Signature · 2 lines
Regards Gui Jianfeng
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 13 lines · Vivek Goyal
Quoted 11 lines · Gui Jianfeng
Vivek, Here are some test results with and without
CONFIG_TRACK_ASYNC_CONTEXTfor V7Mode Normal read | Random read | Normal write | Random write | Direct read | Direct Write
CONFIG_TRACK_ASYNC_CONTEXT=y70,540KiB/s 3,551KiB/s 64,548KiB/s 9,677KiB/s 53,530KiB/s 54,145KiB/s
CONFIG_TRACK_ASYNC_CONTEXT=n71,082KiB/s 3,564KiB/s 66,720KiB/s 9,887KiB/s 51,401KiB/s 55,210KiB/sPerformance +0.7% +0.3% +3.3% +2.1% -4.0% +2.0%
Quoted 7 lines
Strange. Disabling async context tracking should not impact read performance as reads are always sync and don't take async tracking path even if it is enabled. We are instead seeing -4% in direct reads if track async context is disabled.
I would recommend that there can be lot of variance between multiple runs. We should probably run each test 3 times and take some average of that.
Sorry for the late reply.
I tried to test direct reads for 5 times when CONFIG_TRACK_ASYNC_CONTEXT=n and y. I got the following results, and still had the performance variance.
For V7.
1st 2nd 3rd 4th 5th avg
CONFIG_TRACK_ASYNC_CONTEXT=y 58,391KiB/s 58,861KiB/s 58,685KiB/s 59,020KiB/s 58,883KiB/s 58,786KiB/s
CONFIG_TRACK_ASYNC_CONTEXT=n 57,045KiB/s 57,827KiB/s 57,744KiB/s 56,884KiB/s 57,821KiB/s 57,619KiB/s
Performance -2.3% -1.7% -1.6% -3.6% -1.8% -2.0%
Quoted 446 lines
Thanks
VivekQuoted 439 lines
Quoted 434 lines · Vivek Goyal
Quoted 5 lines · Gui Jianfeng
Hi Vivek,
Here are some test results for normal reads and write for IO Controller V7 by fio. Tested with "fairness == 0". It seems performance gets better comparing with V6.
Quoted 8 lines
Mode Normal read | Random read | Normal write | Random write | Direct read | Direct Write
2.6.31-rc1 71,613KiB/s 3,606KiB/s 66,250KiB/s 9,420KiB/s 51,535KiB/s 55,752KiB/s
V7 70,540KiB/s 3,551KiB/s 64,548KiB/s 9,677KiB/s 53,530KiB/s 54,145KiB/s
Performance -1.5% -1.5% -2.6% +2.7% +3.9% -2.9%
Thanks Gui. Can you also try V7 with
CONFIG_TRACK_ASYNC_CONTEXT=n. I tried that and I got better results for buffered writes.In my testing I still see some performance regression for buffered writes which goes away if I disable group io scheduling and just use flat mode.
I will spend more time to find out where it is coming from.
Thanks
VivekQuoted 406 lines
Quoted 402 lines · Vivek Goyal
Hi All,
Here is the V7 of the IO controller patches generated on top of 2.6.31-rc4.
For ease of patching, a consolidated patch is available here.
http://people.redhat.com/~vgoyal/io-controller/io-scheduler-based-io-controller-v7.patch
Previous versions of the patches was posted here.
(V1) http://lkml.org/lkml/2009/3/11/486
(V2) http://lkml.org/lkml/2009/5/5/275
(V3) http://lkml.org/lkml/2009/5/26/472
(V4) http://lkml.org/lkml/2009/6/8/580
(V5) http://lkml.org/lkml/2009/6/19/279
(V6) http://lkml.org/lkml/2009/7/2/369Changes from V6
===============
- Introduced the notion ofgroup_idlingwhere we idle for next request to come from the same group before we expire it. It is along the lines of cfq'sslice_idlething to provide fairness. Switching to group idling now helps in the sense that we don't have to rely whether queue idling was turned on or not by CFQ. It becomes too much of debugging pain with different work loads and different kind of storage media. Introduction ofgroup_idleshould help.- Moved some of the code like dynamic queue idling update, arming queue idling timer, keeping track of average think time etc back to CFQ. With group idling we don't need it now. Reduce the amount of change.
- Enabled cfq's close cooperator functionality in groups. So far this worked only in root group. Now it should work in non-root groups also.
- Got rid of the patch where we calculated disk time based on average disk rate in some circumstances. It was giving bad numbers in early queue deletion cases. Also did not think that it was helping a lot. Remvoed it for the time being.
- Added an experimental patch to map sync requests using bio tracking info and not task context. This is only for noop, deadline and AS.
- Got rid of experimental patch of idling for async queues. Don't think it was helping.
- Got rid of
wait_busyandwait_busy_donelogic from queue. Instead implemented it for groups.- Introduced
oom_ioqto accomodateoom_cfqqchange recently.- Broke-up
elv_init_ioq()fn into smaller functions. It had 7 arguments and looked complicated.- Fixed a bug in
blk_queue_io_group_congested(). Thanks to Munehiro Ikeda.- Merged gui's patch to fix the cgroup file format issue.
- Merged gui's patch to update per group congestion limit when
q->nr_group_requestsis updated.- Fixed a bug where close cooperation will not work if we wait for all the requests to finish from previous queue.
- Fixed group deletion accouting where deletion from idle tree were also appearing in the log.
- Got rid of
busy_rt_queuesinfrastructure.- Got rid of
elv_ioq_request_dispatched(). An helper function just to increment a variable.Limitations
===========- This IO controller provides the bandwidth control at the IO scheduler level (leaf node in stacked hiearchy of logical devices). So there can be cases (depending on configuration) where application does not see proportional BW division at higher logical level device.
LWN has written an article about the issue here.
http://lwn.net/Articles/332839/How to solve the issue of fairness at higher level logical devices
==================================================================
(Do we really need it? That's not where the contention for resources is.)Couple of suggestions have come forward.
- Implement IO control at IO scheduler layer and then with the help of some daemon, adjust the weight on underlying devices dynamiclly, depending on what kind of BW gurantees are to be achieved at higher level logical block devices.
- Also implement a higher level IO controller along with IO scheduler based controller and let user choose one depending on his needs.
A higher level controller does not know about the assumptions/policies of unerldying IO scheduler, hence it has the potential to break down the IO scheduler's policy with-in cgroup. A lower level controller can work with IO scheduler much more closely and efficiently.
Other active IO controller developments
=======================================IO throttling
-------------This is a max bandwidth controller and not the proportional one. Secondly it is a second level controller which can break the IO scheduler's policy/assumtions with-in cgroup.
dm-ioband
---------This is a proportional bandwidth controller implemented as device mapper driver. It is also a second level controller which can break the IO scheduler's policy/assumptions with-in cgroup.
TODO
====
- code cleanups, testing, bug fixing, optimizations, benchmarking etc...Testing
=======I have been able to do some testing as follows. All my testing is with ext3 file system with a SATA drive which supports queue depth of 31.
Test1 (Isolation between two KVM virtual machines)
==================================================
Created two KVM virtual machines. Partitioned a disk on host in two partitions and gave one partition to each virtual machine. Put both the virtual machines in two different cgroup of weight 1000 and 500 each. Virtual machines created ext3 file system on the partitions exported from host and did buffered writes. Host seems writes as synchronous and virtual machine with higher weight gets double the disk time of virtual machine of lower weight. Used deadline scheduler in this test case.Some more details about configuration are in documentation patch.
Test2 (Fairness for synchronous reads)
======================================
- Two dd in two cgroups with cgrop weights 1000 and 500. Ran two "dd" in those cgroups (With CFQ scheduler and /sys/block/<device>/queue/fairness = 1)Higher weight dd finishes first and at that point of time my script takes care of reading cgroup files io.disk_time and io.disk_sectors for both the groups and display the results.
dd if=/mnt/$BLOCKDEV/zerofile1 of=/dev/null &
dd if=/mnt/$BLOCKDEV/zerofile2 of=/dev/null &234179072 bytes (234 MB) copied, 3.9065 s, 59.9 MB/s 234179072 bytes (234 MB) copied, 5.19232 s, 45.1 MB/s
group1 time=8 16 2471 group1 sectors=8 16 457840 group2 time=8 16 1220 group2 sectors=8 16 225736
First two fields in time and sectors statistics represent major and minor number of the device. Third field represents disk time in milliseconds and number of sectors transferred respectively.
This patchset tries to provide fairness in terms of disk time received. group1 got almost double of group2 disk time (At the time of first dd finish). These time and sectors statistics can be read using io.disk_time and io.disk_sector files in cgroup. More about it in documentation file.
Test3 (Reader Vs Buffered Writes)
================================
Buffered writes can be problematic and can overwhelm readers, especially with noop and deadline. IO controller can provide isolation between readers and buffered (async) writers.First I ran the test without io controller to see the severity of the issue. Ran a hostile writer and then after 10 seconds started a reader and then monitored the completion time of reader. Reader reads a 256 MB file. Tested this with noop scheduler.
sample script
------------
sync
echo 3 >/proc/sys/vm/drop_caches
time dd if=/dev/zero of=/mnt/sdb/reader-writer-zerofile bs=4K count=2097152 conv=fdatasync &
sleep 10
time dd if=/mnt/sdb/256M-file of=/dev/null &Results
-------
8589934592 bytes (8.6 GB) copied, 106.045 s, 81.0 MB/s (Writer) 268435456 bytes (268 MB) copied, 96.5237 s, 2.8 MB/s (Reader)Now it was time to test io controller whether it can provide isolation between readers and writers with noop. I created two cgroups of weight 1000 each and put reader in group1 and writer in group 2 and ran the test again. Upon comletion of reader, my scripts read io.dis_time and io.disk_group cgroup files to get an estimate how much disk time each group got and how many sectors each group did IO for.
For more accurate accounting of disk time for buffered writes with queuing hardware I had to set /sys/block/<disk>/queue/iosched/fairness to "1".
sample script
-------------
echo $$ > /cgroup/bfqio/test2/tasks
dd if=/dev/zero of=/mnt/$BLOCKDEV/testzerofile bs=4K count=2097152 & sleep 10
echo noop >/sys/block/$BLOCKDEV/queue/scheduler
echo 1 >/sys/block/$BLOCKDEV/queue/iosched/fairnessecho $$ > /cgroup/bfqio/test1/tasks
dd if=/mnt/$BLOCKDEV/256M-file of=/dev/null &
wait $!
# Some code for reading cgroup files upon completion of reader.
-------------------------Results
=======
268435456 bytes (268 MB) copied, 6.65819 s, 40.3 MB/s (Reader)group1 time=8 16 3063 group1 sectors=8 16 524808
group2 time=8 16 3071 group2 sectors=8 16 441752Note, reader finishes now much lesser time and both group1 and group2 got almost 3 seconds of disk time. Hence io-controller provides isolation from buffered writes.
Test4 (AIO)
===========AIO reads
-----------
Set up two fio, AIO read jobs in two cgroup with weight 1000 and 500 respectively. I am using cfq scheduler. Following are some lines from my test script.---------------------------------------------------------------
echo 1000 > /cgroup/bfqio/test1/io.weight
echo 500 > /cgroup/bfqio/test2/io.weightfio_args="--ioengine=libaio --rw=read --size=512M --direct=1" echo 1 >
/sys/block/$BLOCKDEV/queue/iosched/fairnessecho $$ > /cgroup/bfqio/test1/tasks
fio $fio_args --name=test1 --directory=/mnt/$BLOCKDEV/fio1/ --output=/mnt/$BLOCKDEV/fio1/test1.log --exec_postrun="../read-and-display-group-stats.sh $maj_dev $minor_dev" &echo $$ > /cgroup/bfqio/test2/tasks
fio $fio_args --name=test2 --directory=/mnt/$BLOCKDEV/fio2/ --output=/mnt/$BLOCKDEV/fio2/test2.log &
----------------------------------------------------------------test1 and test2 are two groups with weight 1000 and 500 respectively. "
read-and-display-group-stats.sh" is one small script which reads the test1 and test2 cgroup files to determine how much disk time each group got till first fio job finished.Results
------
test1 statistics: time=8 16 22403 sectors=8 16 1049640
test2 statistics: time=8 16 11400 sectors=8 16 552864Above shows that by the time first fio (higher weight), finished, group test1 got 22403 ms of disk time and group test2 got 11400 ms of disk time. similarly the statistics for number of sectors transferred are also shown.
Note that disk time given to group test1 is almost double of group2 disk time.
AIO writes
----------
Set up two fio, AIO direct write jobs in two cgroup with weight 1000 and 500 respectively. I am using cfq scheduler. Following are some lines from my test script.------------------------------------------------
echo 1000 > /cgroup/bfqio/test1/io.weight
echo 500 > /cgroup/bfqio/test2/io.weight fio_args="--ioengine=libaio --rw=write --size=512M --direct=1"echo 1 >
/sys/block/$BLOCKDEV/queue/iosched/fairnessecho $$ > /cgroup/bfqio/test1/tasks
fio $fio_args --name=test1 --directory=/mnt/$BLOCKDEV/fio1/ --output=/mnt/$BLOCKDEV/fio1/test1.log --exec_postrun="../read-and-display-group-stats.sh $maj_dev $minor_dev" &echo $$ > /cgroup/bfqio/test2/tasks
fio $fio_args --name=test2 --directory=/mnt/$BLOCKDEV/fio2/ --output=/mnt/$BLOCKDEV/fio2/test2.log &
-------------------------------------------------test1 and test2 are two groups with weight 1000 and 500 respectively. "
read-and-display-group-stats.sh" is one small script which reads the test1 and test2 cgroup files to determine how much disk time each group got till first fio job finished.Following are the results.
test1 statistics: time=8 16 29085 sectors=8 16 1049656
test2 statistics: time=8 16 14652 sectors=8 16 516728Above shows that by the time first fio (higher weight), finished, group test1 got 28085 ms of disk time and group test2 got 14652 ms of disk time. similarly the statistics for number of sectors transferred are also shown.
Note that disk time given to group test1 is almost double of group2 disk time.
Test5 (Fairness for async writes, Buffered Write Vs Buffered Write)
===================================================================
Fairness for async writes is tricky and biggest reason is that async writes are cached in higher layers (page cahe) as well as possibly in file system layer also (btrfs, xfs etc), and are dispatched to lower layers not necessarily in proportional manner.For example, consider two dd threads reading
/dev/zeroas input file and doing writes of huge files. Very soon we will crossvm_dirty_ratioand dd thread will be forced to write out some pages to disk before more pages can be dirtied. But not necessarily dirty pages of same thread are picked. It can very well pick the inode of lesser priority dd thread and do some writeout. So effectively higher weight dd is doing writeouts of lower weight dd pages and we don't see service differentation.IOW, the core problem with async write fairness is that higher weight thread does not throw enought IO traffic at IO controller to keep the queue continuously backlogged. In my testing, there are many .2 to .8 second intervals where higher weight queue is empty and in that duration lower weight queue get lots of job done giving the impression that there was no service differentiation.
In summary, from IO controller point of view async writes support is there. Because page cache has not been designed in such a manner that higher prio/weight writer can do more write out as compared to lower prio/weight writer, gettting service differentiation is hard and it is visible in some cases and not visible in some cases.
Do we really care that much for fairness among two writer cgroups? One can choose to do direct writes or sync writes if fairness for writes really matters for him.
Following is the only case where it is hard to ensure fairness between cgroups.
- Buffered writes Vs Buffered Writes.
So to test async writes I created two partitions on a disk and created ext3 file systems on both the partitions. Also created two cgroups and generated lots of write traffic in two cgroups (50 fio threads) and watched the disk time statistics in respective cgroups at the interval of 2 seconds. Thanks to ryo tsuruta for the test case.
*****************************************************************
sync
echo 3 >/proc/sys/vm/drop_cachesfio_args="--size=64m --rw=write --numjobs=50 --group_reporting"
echo $$ > /cgroup/bfqio/test1/tasks
fio $fio_args --name=test1 --directory=/mnt/sdd1/fio/ --output=/mnt/sdd1/fio/test1.log &echo $$ > /cgroup/bfqio/test2/tasks
fio $fio_args --name=test2 --directory=/mnt/sdd2/fio/ --output=/mnt/sdd2/fio/test2.log &
***********************************************************************And watched the disk time and sector statistics for the both the cgroups every 2 seconds using a script. How is snippet from output.
test1 statistics: time=8 48 1315 sectors=8 48 55776 dq=8 48 1
test2 statistics: time=8 48 633 sectors=8 48 14720 dq=8 48 2test1 statistics: time=8 48 5586 sectors=8 48 339064 dq=8 48 2
test2 statistics: time=8 48 2985 sectors=8 48 146656 dq=8 48 3test1 statistics: time=8 48 9935 sectors=8 48 628728 dq=8 48 3
test2 statistics: time=8 48 5265 sectors=8 48 278688 dq=8 48 4test1 statistics: time=8 48 14156 sectors=8 48 932488 dq=8 48 6
test2 statistics: time=8 48 7646 sectors=8 48 412704 dq=8 48 7test1 statistics: time=8 48 18141 sectors=8 48 1231488 dq=8 48 10
test2 statistics: time=8 48 9820 sectors=8 48 548400 dq=8 48 8test1 statistics: time=8 48 21953 sectors=8 48 1485632 dq=8 48 13
test2 statistics: time=8 48 12394 sectors=8 48 698288 dq=8 48 10test1 statistics: time=8 48 25167 sectors=8 48 1705264 dq=8 48 13
test2 statistics: time=8 48 14042 sectors=8 48 817808 dq=8 48 10First two fields in time and sectors statistics represent major and minor number of the device. Third field represents disk time in milliseconds and number of sectors transferred respectively.
So disk time consumed by group1 is almost double of group2 in this case.
Your feedback is welcome.
Thanks
VivekSignature · 2 lines
Regards Gui JianfengSignature · 2 lines
Regards Gui Jianfeng
Signature · 2 lines
Regards Gui Jianfeng
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 11 lines · Gui Jianfeng
Vivek, Here are some test results with and without
CONFIG_TRACK_ASYNC_CONTEXTfor V7Mode Normal read | Random read | Normal write | Random write | Direct read | Direct Write
CONFIG_TRACK_ASYNC_CONTEXT=y70,540KiB/s 3,551KiB/s 64,548KiB/s 9,677KiB/s 53,530KiB/s 54,145KiB/s
CONFIG_TRACK_ASYNC_CONTEXT=n71,082KiB/s 3,564KiB/s 66,720KiB/s 9,887KiB/s 51,401KiB/s 55,210KiB/sPerformance +0.7% +0.3% +3.3% +2.1% -4.0% +2.0%
Strange. Disabling async context tracking should not impact read performance as reads are always sync and don't take async tracking path even if it is enabled. We are instead seeing -4% in direct reads if track async context is disabled.
I would recommend that there can be lot of variance between multiple runs. We should probably run each test 3 times and take some average of that.
Thanks
Vivek
Quoted 442 lines
Quoted 436 lines · Vivek Goyal
Quoted 5 lines · Gui Jianfeng
Hi Vivek,
Here are some test results for normal reads and write for IO Controller V7 by fio. Tested with "fairness == 0". It seems performance gets better comparing with V6.
Quoted 8 lines
Mode Normal read | Random read | Normal write | Random write | Direct read | Direct Write
2.6.31-rc1 71,613KiB/s 3,606KiB/s 66,250KiB/s 9,420KiB/s 51,535KiB/s 55,752KiB/s
V7 70,540KiB/s 3,551KiB/s 64,548KiB/s 9,677KiB/s 53,530KiB/s 54,145KiB/s
Performance -1.5% -1.5% -2.6% +2.7% +3.9% -2.9%
Thanks Gui. Can you also try V7 with
CONFIG_TRACK_ASYNC_CONTEXT=n. I tried that and I got better results for buffered writes.In my testing I still see some performance regression for buffered writes which goes away if I disable group io scheduling and just use flat mode.
I will spend more time to find out where it is coming from.
Thanks
VivekQuoted 406 lines
Quoted 402 lines · Vivek Goyal
Hi All,
Here is the V7 of the IO controller patches generated on top of 2.6.31-rc4.
For ease of patching, a consolidated patch is available here.
http://people.redhat.com/~vgoyal/io-controller/io-scheduler-based-io-controller-v7.patch
Previous versions of the patches was posted here.
(V1) http://lkml.org/lkml/2009/3/11/486
(V2) http://lkml.org/lkml/2009/5/5/275
(V3) http://lkml.org/lkml/2009/5/26/472
(V4) http://lkml.org/lkml/2009/6/8/580
(V5) http://lkml.org/lkml/2009/6/19/279
(V6) http://lkml.org/lkml/2009/7/2/369Changes from V6
===============
- Introduced the notion ofgroup_idlingwhere we idle for next request to come from the same group before we expire it. It is along the lines of cfq'sslice_idlething to provide fairness. Switching to group idling now helps in the sense that we don't have to rely whether queue idling was turned on or not by CFQ. It becomes too much of debugging pain with different work loads and different kind of storage media. Introduction ofgroup_idleshould help.- Moved some of the code like dynamic queue idling update, arming queue idling timer, keeping track of average think time etc back to CFQ. With group idling we don't need it now. Reduce the amount of change.
- Enabled cfq's close cooperator functionality in groups. So far this worked only in root group. Now it should work in non-root groups also.
- Got rid of the patch where we calculated disk time based on average disk rate in some circumstances. It was giving bad numbers in early queue deletion cases. Also did not think that it was helping a lot. Remvoed it for the time being.
- Added an experimental patch to map sync requests using bio tracking info and not task context. This is only for noop, deadline and AS.
- Got rid of experimental patch of idling for async queues. Don't think it was helping.
- Got rid of
wait_busyandwait_busy_donelogic from queue. Instead implemented it for groups.- Introduced
oom_ioqto accomodateoom_cfqqchange recently.- Broke-up
elv_init_ioq()fn into smaller functions. It had 7 arguments and looked complicated.- Fixed a bug in
blk_queue_io_group_congested(). Thanks to Munehiro Ikeda.- Merged gui's patch to fix the cgroup file format issue.
- Merged gui's patch to update per group congestion limit when
q->nr_group_requestsis updated.- Fixed a bug where close cooperation will not work if we wait for all the requests to finish from previous queue.
- Fixed group deletion accouting where deletion from idle tree were also appearing in the log.
- Got rid of
busy_rt_queuesinfrastructure.- Got rid of
elv_ioq_request_dispatched(). An helper function just to increment a variable.Limitations
===========- This IO controller provides the bandwidth control at the IO scheduler level (leaf node in stacked hiearchy of logical devices). So there can be cases (depending on configuration) where application does not see proportional BW division at higher logical level device.
LWN has written an article about the issue here.
http://lwn.net/Articles/332839/How to solve the issue of fairness at higher level logical devices
==================================================================
(Do we really need it? That's not where the contention for resources is.)Couple of suggestions have come forward.
- Implement IO control at IO scheduler layer and then with the help of some daemon, adjust the weight on underlying devices dynamiclly, depending on what kind of BW gurantees are to be achieved at higher level logical block devices.
- Also implement a higher level IO controller along with IO scheduler based controller and let user choose one depending on his needs.
A higher level controller does not know about the assumptions/policies of unerldying IO scheduler, hence it has the potential to break down the IO scheduler's policy with-in cgroup. A lower level controller can work with IO scheduler much more closely and efficiently.
Other active IO controller developments
=======================================IO throttling
-------------This is a max bandwidth controller and not the proportional one. Secondly it is a second level controller which can break the IO scheduler's policy/assumtions with-in cgroup.
dm-ioband
---------This is a proportional bandwidth controller implemented as device mapper driver. It is also a second level controller which can break the IO scheduler's policy/assumptions with-in cgroup.
TODO
====
- code cleanups, testing, bug fixing, optimizations, benchmarking etc...Testing
=======I have been able to do some testing as follows. All my testing is with ext3 file system with a SATA drive which supports queue depth of 31.
Test1 (Isolation between two KVM virtual machines)
==================================================
Created two KVM virtual machines. Partitioned a disk on host in two partitions and gave one partition to each virtual machine. Put both the virtual machines in two different cgroup of weight 1000 and 500 each. Virtual machines created ext3 file system on the partitions exported from host and did buffered writes. Host seems writes as synchronous and virtual machine with higher weight gets double the disk time of virtual machine of lower weight. Used deadline scheduler in this test case.Some more details about configuration are in documentation patch.
Test2 (Fairness for synchronous reads)
======================================
- Two dd in two cgroups with cgrop weights 1000 and 500. Ran two "dd" in those cgroups (With CFQ scheduler and /sys/block/<device>/queue/fairness = 1)Higher weight dd finishes first and at that point of time my script takes care of reading cgroup files io.disk_time and io.disk_sectors for both the groups and display the results.
dd if=/mnt/$BLOCKDEV/zerofile1 of=/dev/null &
dd if=/mnt/$BLOCKDEV/zerofile2 of=/dev/null &234179072 bytes (234 MB) copied, 3.9065 s, 59.9 MB/s 234179072 bytes (234 MB) copied, 5.19232 s, 45.1 MB/s
group1 time=8 16 2471 group1 sectors=8 16 457840 group2 time=8 16 1220 group2 sectors=8 16 225736
First two fields in time and sectors statistics represent major and minor number of the device. Third field represents disk time in milliseconds and number of sectors transferred respectively.
This patchset tries to provide fairness in terms of disk time received. group1 got almost double of group2 disk time (At the time of first dd finish). These time and sectors statistics can be read using io.disk_time and io.disk_sector files in cgroup. More about it in documentation file.
Test3 (Reader Vs Buffered Writes)
================================
Buffered writes can be problematic and can overwhelm readers, especially with noop and deadline. IO controller can provide isolation between readers and buffered (async) writers.First I ran the test without io controller to see the severity of the issue. Ran a hostile writer and then after 10 seconds started a reader and then monitored the completion time of reader. Reader reads a 256 MB file. Tested this with noop scheduler.
sample script
------------
sync
echo 3 >/proc/sys/vm/drop_caches
time dd if=/dev/zero of=/mnt/sdb/reader-writer-zerofile bs=4K count=2097152 conv=fdatasync &
sleep 10
time dd if=/mnt/sdb/256M-file of=/dev/null &Results
-------
8589934592 bytes (8.6 GB) copied, 106.045 s, 81.0 MB/s (Writer) 268435456 bytes (268 MB) copied, 96.5237 s, 2.8 MB/s (Reader)Now it was time to test io controller whether it can provide isolation between readers and writers with noop. I created two cgroups of weight 1000 each and put reader in group1 and writer in group 2 and ran the test again. Upon comletion of reader, my scripts read io.dis_time and io.disk_group cgroup files to get an estimate how much disk time each group got and how many sectors each group did IO for.
For more accurate accounting of disk time for buffered writes with queuing hardware I had to set /sys/block/<disk>/queue/iosched/fairness to "1".
sample script
-------------
echo $$ > /cgroup/bfqio/test2/tasks
dd if=/dev/zero of=/mnt/$BLOCKDEV/testzerofile bs=4K count=2097152 & sleep 10
echo noop >/sys/block/$BLOCKDEV/queue/scheduler
echo 1 >/sys/block/$BLOCKDEV/queue/iosched/fairnessecho $$ > /cgroup/bfqio/test1/tasks
dd if=/mnt/$BLOCKDEV/256M-file of=/dev/null &
wait $!
# Some code for reading cgroup files upon completion of reader.
-------------------------Results
=======
268435456 bytes (268 MB) copied, 6.65819 s, 40.3 MB/s (Reader)group1 time=8 16 3063 group1 sectors=8 16 524808
group2 time=8 16 3071 group2 sectors=8 16 441752Note, reader finishes now much lesser time and both group1 and group2 got almost 3 seconds of disk time. Hence io-controller provides isolation from buffered writes.
Test4 (AIO)
===========AIO reads
-----------
Set up two fio, AIO read jobs in two cgroup with weight 1000 and 500 respectively. I am using cfq scheduler. Following are some lines from my test script.---------------------------------------------------------------
echo 1000 > /cgroup/bfqio/test1/io.weight
echo 500 > /cgroup/bfqio/test2/io.weightfio_args="--ioengine=libaio --rw=read --size=512M --direct=1" echo 1 >
/sys/block/$BLOCKDEV/queue/iosched/fairnessecho $$ > /cgroup/bfqio/test1/tasks
fio $fio_args --name=test1 --directory=/mnt/$BLOCKDEV/fio1/ --output=/mnt/$BLOCKDEV/fio1/test1.log --exec_postrun="../read-and-display-group-stats.sh $maj_dev $minor_dev" &echo $$ > /cgroup/bfqio/test2/tasks
fio $fio_args --name=test2 --directory=/mnt/$BLOCKDEV/fio2/ --output=/mnt/$BLOCKDEV/fio2/test2.log &
----------------------------------------------------------------test1 and test2 are two groups with weight 1000 and 500 respectively. "
read-and-display-group-stats.sh" is one small script which reads the test1 and test2 cgroup files to determine how much disk time each group got till first fio job finished.Results
------
test1 statistics: time=8 16 22403 sectors=8 16 1049640
test2 statistics: time=8 16 11400 sectors=8 16 552864Above shows that by the time first fio (higher weight), finished, group test1 got 22403 ms of disk time and group test2 got 11400 ms of disk time. similarly the statistics for number of sectors transferred are also shown.
Note that disk time given to group test1 is almost double of group2 disk time.
AIO writes
----------
Set up two fio, AIO direct write jobs in two cgroup with weight 1000 and 500 respectively. I am using cfq scheduler. Following are some lines from my test script.------------------------------------------------
echo 1000 > /cgroup/bfqio/test1/io.weight
echo 500 > /cgroup/bfqio/test2/io.weight fio_args="--ioengine=libaio --rw=write --size=512M --direct=1"echo 1 >
/sys/block/$BLOCKDEV/queue/iosched/fairnessecho $$ > /cgroup/bfqio/test1/tasks
fio $fio_args --name=test1 --directory=/mnt/$BLOCKDEV/fio1/ --output=/mnt/$BLOCKDEV/fio1/test1.log --exec_postrun="../read-and-display-group-stats.sh $maj_dev $minor_dev" &echo $$ > /cgroup/bfqio/test2/tasks
fio $fio_args --name=test2 --directory=/mnt/$BLOCKDEV/fio2/ --output=/mnt/$BLOCKDEV/fio2/test2.log &
-------------------------------------------------test1 and test2 are two groups with weight 1000 and 500 respectively. "
read-and-display-group-stats.sh" is one small script which reads the test1 and test2 cgroup files to determine how much disk time each group got till first fio job finished.Following are the results.
test1 statistics: time=8 16 29085 sectors=8 16 1049656
test2 statistics: time=8 16 14652 sectors=8 16 516728Above shows that by the time first fio (higher weight), finished, group test1 got 28085 ms of disk time and group test2 got 14652 ms of disk time. similarly the statistics for number of sectors transferred are also shown.
Note that disk time given to group test1 is almost double of group2 disk time.
Test5 (Fairness for async writes, Buffered Write Vs Buffered Write)
===================================================================
Fairness for async writes is tricky and biggest reason is that async writes are cached in higher layers (page cahe) as well as possibly in file system layer also (btrfs, xfs etc), and are dispatched to lower layers not necessarily in proportional manner.For example, consider two dd threads reading
/dev/zeroas input file and doing writes of huge files. Very soon we will crossvm_dirty_ratioand dd thread will be forced to write out some pages to disk before more pages can be dirtied. But not necessarily dirty pages of same thread are picked. It can very well pick the inode of lesser priority dd thread and do some writeout. So effectively higher weight dd is doing writeouts of lower weight dd pages and we don't see service differentation.IOW, the core problem with async write fairness is that higher weight thread does not throw enought IO traffic at IO controller to keep the queue continuously backlogged. In my testing, there are many .2 to .8 second intervals where higher weight queue is empty and in that duration lower weight queue get lots of job done giving the impression that there was no service differentiation.
In summary, from IO controller point of view async writes support is there. Because page cache has not been designed in such a manner that higher prio/weight writer can do more write out as compared to lower prio/weight writer, gettting service differentiation is hard and it is visible in some cases and not visible in some cases.
Do we really care that much for fairness among two writer cgroups? One can choose to do direct writes or sync writes if fairness for writes really matters for him.
Following is the only case where it is hard to ensure fairness between cgroups.
- Buffered writes Vs Buffered Writes.
So to test async writes I created two partitions on a disk and created ext3 file systems on both the partitions. Also created two cgroups and generated lots of write traffic in two cgroups (50 fio threads) and watched the disk time statistics in respective cgroups at the interval of 2 seconds. Thanks to ryo tsuruta for the test case.
*****************************************************************
sync
echo 3 >/proc/sys/vm/drop_cachesfio_args="--size=64m --rw=write --numjobs=50 --group_reporting"
echo $$ > /cgroup/bfqio/test1/tasks
fio $fio_args --name=test1 --directory=/mnt/sdd1/fio/ --output=/mnt/sdd1/fio/test1.log &echo $$ > /cgroup/bfqio/test2/tasks
fio $fio_args --name=test2 --directory=/mnt/sdd2/fio/ --output=/mnt/sdd2/fio/test2.log &
***********************************************************************And watched the disk time and sector statistics for the both the cgroups every 2 seconds using a script. How is snippet from output.
test1 statistics: time=8 48 1315 sectors=8 48 55776 dq=8 48 1
test2 statistics: time=8 48 633 sectors=8 48 14720 dq=8 48 2test1 statistics: time=8 48 5586 sectors=8 48 339064 dq=8 48 2
test2 statistics: time=8 48 2985 sectors=8 48 146656 dq=8 48 3test1 statistics: time=8 48 9935 sectors=8 48 628728 dq=8 48 3
test2 statistics: time=8 48 5265 sectors=8 48 278688 dq=8 48 4test1 statistics: time=8 48 14156 sectors=8 48 932488 dq=8 48 6
test2 statistics: time=8 48 7646 sectors=8 48 412704 dq=8 48 7test1 statistics: time=8 48 18141 sectors=8 48 1231488 dq=8 48 10
test2 statistics: time=8 48 9820 sectors=8 48 548400 dq=8 48 8test1 statistics: time=8 48 21953 sectors=8 48 1485632 dq=8 48 13
test2 statistics: time=8 48 12394 sectors=8 48 698288 dq=8 48 10test1 statistics: time=8 48 25167 sectors=8 48 1705264 dq=8 48 13
test2 statistics: time=8 48 14042 sectors=8 48 817808 dq=8 48 10First two fields in time and sectors statistics represent major and minor number of the device. Third field represents disk time in milliseconds and number of sectors transferred respectively.
So disk time consumed by group1 is almost double of group2 in this case.
Your feedback is welcome.
Thanks
VivekSignature · 2 lines
Regards Gui JianfengSignature · 2 lines
Regards Gui Jianfeng
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 5 lines · Gui Jianfeng
Hi Vivek,
Here are some test results for normal reads and write for IO Controller V7 by fio. Tested with "fairness == 0". It seems performance gets better comparing with V6.
Quoted 8 lines
Mode Normal read | Random read | Normal write | Random write | Direct read | Direct Write
2.6.31-rc1 71,613KiB/s 3,606KiB/s 66,250KiB/s 9,420KiB/s 51,535KiB/s 55,752KiB/s
V7 70,540KiB/s 3,551KiB/s 64,548KiB/s 9,677KiB/s 53,530KiB/s 54,145KiB/s
Performance -1.5% -1.5% -2.6% +2.7% +3.9% -2.9%
Thanks Gui. Can you also try V7 with CONFIG_TRACK_ASYNC_CONTEXT=n. I tried that and I got better results for buffered writes.
In my testing I still see some performance regression for buffered writes which goes away if I disable group io scheduling and just use flat mode.
I will spend more time to find out where it is coming from.
Thanks
Vivek
Quoted 408 lines
Quoted 402 lines · Vivek Goyal
Hi All,
Here is the V7 of the IO controller patches generated on top of 2.6.31-rc4.
For ease of patching, a consolidated patch is available here.
http://people.redhat.com/~vgoyal/io-controller/io-scheduler-based-io-controller-v7.patch
Previous versions of the patches was posted here.
(V1) http://lkml.org/lkml/2009/3/11/486
(V2) http://lkml.org/lkml/2009/5/5/275
(V3) http://lkml.org/lkml/2009/5/26/472
(V4) http://lkml.org/lkml/2009/6/8/580
(V5) http://lkml.org/lkml/2009/6/19/279
(V6) http://lkml.org/lkml/2009/7/2/369Changes from V6
===============
- Introduced the notion ofgroup_idlingwhere we idle for next request to come from the same group before we expire it. It is along the lines of cfq'sslice_idlething to provide fairness. Switching to group idling now helps in the sense that we don't have to rely whether queue idling was turned on or not by CFQ. It becomes too much of debugging pain with different work loads and different kind of storage media. Introduction ofgroup_idleshould help.- Moved some of the code like dynamic queue idling update, arming queue idling timer, keeping track of average think time etc back to CFQ. With group idling we don't need it now. Reduce the amount of change.
- Enabled cfq's close cooperator functionality in groups. So far this worked only in root group. Now it should work in non-root groups also.
- Got rid of the patch where we calculated disk time based on average disk rate in some circumstances. It was giving bad numbers in early queue deletion cases. Also did not think that it was helping a lot. Remvoed it for the time being.
- Added an experimental patch to map sync requests using bio tracking info and not task context. This is only for noop, deadline and AS.
- Got rid of experimental patch of idling for async queues. Don't think it was helping.
- Got rid of
wait_busyandwait_busy_donelogic from queue. Instead implemented it for groups.- Introduced
oom_ioqto accomodateoom_cfqqchange recently.- Broke-up
elv_init_ioq()fn into smaller functions. It had 7 arguments and looked complicated.- Fixed a bug in
blk_queue_io_group_congested(). Thanks to Munehiro Ikeda.- Merged gui's patch to fix the cgroup file format issue.
- Merged gui's patch to update per group congestion limit when
q->nr_group_requestsis updated.- Fixed a bug where close cooperation will not work if we wait for all the requests to finish from previous queue.
- Fixed group deletion accouting where deletion from idle tree were also appearing in the log.
- Got rid of
busy_rt_queuesinfrastructure.- Got rid of
elv_ioq_request_dispatched(). An helper function just to increment a variable.Limitations
===========- This IO controller provides the bandwidth control at the IO scheduler level (leaf node in stacked hiearchy of logical devices). So there can be cases (depending on configuration) where application does not see proportional BW division at higher logical level device.
LWN has written an article about the issue here.
http://lwn.net/Articles/332839/How to solve the issue of fairness at higher level logical devices
==================================================================
(Do we really need it? That's not where the contention for resources is.)Couple of suggestions have come forward.
- Implement IO control at IO scheduler layer and then with the help of some daemon, adjust the weight on underlying devices dynamiclly, depending on what kind of BW gurantees are to be achieved at higher level logical block devices.
- Also implement a higher level IO controller along with IO scheduler based controller and let user choose one depending on his needs.
A higher level controller does not know about the assumptions/policies of unerldying IO scheduler, hence it has the potential to break down the IO scheduler's policy with-in cgroup. A lower level controller can work with IO scheduler much more closely and efficiently.
Other active IO controller developments
=======================================IO throttling
-------------This is a max bandwidth controller and not the proportional one. Secondly it is a second level controller which can break the IO scheduler's policy/assumtions with-in cgroup.
dm-ioband
---------This is a proportional bandwidth controller implemented as device mapper driver. It is also a second level controller which can break the IO scheduler's policy/assumptions with-in cgroup.
TODO
====
- code cleanups, testing, bug fixing, optimizations, benchmarking etc...Testing
=======I have been able to do some testing as follows. All my testing is with ext3 file system with a SATA drive which supports queue depth of 31.
Test1 (Isolation between two KVM virtual machines)
==================================================
Created two KVM virtual machines. Partitioned a disk on host in two partitions and gave one partition to each virtual machine. Put both the virtual machines in two different cgroup of weight 1000 and 500 each. Virtual machines created ext3 file system on the partitions exported from host and did buffered writes. Host seems writes as synchronous and virtual machine with higher weight gets double the disk time of virtual machine of lower weight. Used deadline scheduler in this test case.Some more details about configuration are in documentation patch.
Test2 (Fairness for synchronous reads)
======================================
- Two dd in two cgroups with cgrop weights 1000 and 500. Ran two "dd" in those cgroups (With CFQ scheduler and /sys/block/<device>/queue/fairness = 1)Higher weight dd finishes first and at that point of time my script takes care of reading cgroup files io.disk_time and io.disk_sectors for both the groups and display the results.
dd if=/mnt/$BLOCKDEV/zerofile1 of=/dev/null &
dd if=/mnt/$BLOCKDEV/zerofile2 of=/dev/null &234179072 bytes (234 MB) copied, 3.9065 s, 59.9 MB/s 234179072 bytes (234 MB) copied, 5.19232 s, 45.1 MB/s
group1 time=8 16 2471 group1 sectors=8 16 457840 group2 time=8 16 1220 group2 sectors=8 16 225736
First two fields in time and sectors statistics represent major and minor number of the device. Third field represents disk time in milliseconds and number of sectors transferred respectively.
This patchset tries to provide fairness in terms of disk time received. group1 got almost double of group2 disk time (At the time of first dd finish). These time and sectors statistics can be read using io.disk_time and io.disk_sector files in cgroup. More about it in documentation file.
Test3 (Reader Vs Buffered Writes)
================================
Buffered writes can be problematic and can overwhelm readers, especially with noop and deadline. IO controller can provide isolation between readers and buffered (async) writers.First I ran the test without io controller to see the severity of the issue. Ran a hostile writer and then after 10 seconds started a reader and then monitored the completion time of reader. Reader reads a 256 MB file. Tested this with noop scheduler.
sample script
------------
sync
echo 3 >/proc/sys/vm/drop_caches
time dd if=/dev/zero of=/mnt/sdb/reader-writer-zerofile bs=4K count=2097152 conv=fdatasync &
sleep 10
time dd if=/mnt/sdb/256M-file of=/dev/null &Results
-------
8589934592 bytes (8.6 GB) copied, 106.045 s, 81.0 MB/s (Writer) 268435456 bytes (268 MB) copied, 96.5237 s, 2.8 MB/s (Reader)Now it was time to test io controller whether it can provide isolation between readers and writers with noop. I created two cgroups of weight 1000 each and put reader in group1 and writer in group 2 and ran the test again. Upon comletion of reader, my scripts read io.dis_time and io.disk_group cgroup files to get an estimate how much disk time each group got and how many sectors each group did IO for.
For more accurate accounting of disk time for buffered writes with queuing hardware I had to set /sys/block/<disk>/queue/iosched/fairness to "1".
sample script
-------------
echo $$ > /cgroup/bfqio/test2/tasks
dd if=/dev/zero of=/mnt/$BLOCKDEV/testzerofile bs=4K count=2097152 & sleep 10
echo noop >/sys/block/$BLOCKDEV/queue/scheduler
echo 1 >/sys/block/$BLOCKDEV/queue/iosched/fairnessecho $$ > /cgroup/bfqio/test1/tasks
dd if=/mnt/$BLOCKDEV/256M-file of=/dev/null &
wait $!
# Some code for reading cgroup files upon completion of reader.
-------------------------Results
=======
268435456 bytes (268 MB) copied, 6.65819 s, 40.3 MB/s (Reader)group1 time=8 16 3063 group1 sectors=8 16 524808
group2 time=8 16 3071 group2 sectors=8 16 441752Note, reader finishes now much lesser time and both group1 and group2 got almost 3 seconds of disk time. Hence io-controller provides isolation from buffered writes.
Test4 (AIO)
===========AIO reads
-----------
Set up two fio, AIO read jobs in two cgroup with weight 1000 and 500 respectively. I am using cfq scheduler. Following are some lines from my test script.---------------------------------------------------------------
echo 1000 > /cgroup/bfqio/test1/io.weight
echo 500 > /cgroup/bfqio/test2/io.weightfio_args="--ioengine=libaio --rw=read --size=512M --direct=1" echo 1 >
/sys/block/$BLOCKDEV/queue/iosched/fairnessecho $$ > /cgroup/bfqio/test1/tasks
fio $fio_args --name=test1 --directory=/mnt/$BLOCKDEV/fio1/ --output=/mnt/$BLOCKDEV/fio1/test1.log --exec_postrun="../read-and-display-group-stats.sh $maj_dev $minor_dev" &echo $$ > /cgroup/bfqio/test2/tasks
fio $fio_args --name=test2 --directory=/mnt/$BLOCKDEV/fio2/ --output=/mnt/$BLOCKDEV/fio2/test2.log &
----------------------------------------------------------------test1 and test2 are two groups with weight 1000 and 500 respectively. "
read-and-display-group-stats.sh" is one small script which reads the test1 and test2 cgroup files to determine how much disk time each group got till first fio job finished.Results
------
test1 statistics: time=8 16 22403 sectors=8 16 1049640
test2 statistics: time=8 16 11400 sectors=8 16 552864Above shows that by the time first fio (higher weight), finished, group test1 got 22403 ms of disk time and group test2 got 11400 ms of disk time. similarly the statistics for number of sectors transferred are also shown.
Note that disk time given to group test1 is almost double of group2 disk time.
AIO writes
----------
Set up two fio, AIO direct write jobs in two cgroup with weight 1000 and 500 respectively. I am using cfq scheduler. Following are some lines from my test script.------------------------------------------------
echo 1000 > /cgroup/bfqio/test1/io.weight
echo 500 > /cgroup/bfqio/test2/io.weight fio_args="--ioengine=libaio --rw=write --size=512M --direct=1"echo 1 >
/sys/block/$BLOCKDEV/queue/iosched/fairnessecho $$ > /cgroup/bfqio/test1/tasks
fio $fio_args --name=test1 --directory=/mnt/$BLOCKDEV/fio1/ --output=/mnt/$BLOCKDEV/fio1/test1.log --exec_postrun="../read-and-display-group-stats.sh $maj_dev $minor_dev" &echo $$ > /cgroup/bfqio/test2/tasks
fio $fio_args --name=test2 --directory=/mnt/$BLOCKDEV/fio2/ --output=/mnt/$BLOCKDEV/fio2/test2.log &
-------------------------------------------------test1 and test2 are two groups with weight 1000 and 500 respectively. "
read-and-display-group-stats.sh" is one small script which reads the test1 and test2 cgroup files to determine how much disk time each group got till first fio job finished.Following are the results.
test1 statistics: time=8 16 29085 sectors=8 16 1049656
test2 statistics: time=8 16 14652 sectors=8 16 516728Above shows that by the time first fio (higher weight), finished, group test1 got 28085 ms of disk time and group test2 got 14652 ms of disk time. similarly the statistics for number of sectors transferred are also shown.
Note that disk time given to group test1 is almost double of group2 disk time.
Test5 (Fairness for async writes, Buffered Write Vs Buffered Write)
===================================================================
Fairness for async writes is tricky and biggest reason is that async writes are cached in higher layers (page cahe) as well as possibly in file system layer also (btrfs, xfs etc), and are dispatched to lower layers not necessarily in proportional manner.For example, consider two dd threads reading
/dev/zeroas input file and doing writes of huge files. Very soon we will crossvm_dirty_ratioand dd thread will be forced to write out some pages to disk before more pages can be dirtied. But not necessarily dirty pages of same thread are picked. It can very well pick the inode of lesser priority dd thread and do some writeout. So effectively higher weight dd is doing writeouts of lower weight dd pages and we don't see service differentation.IOW, the core problem with async write fairness is that higher weight thread does not throw enought IO traffic at IO controller to keep the queue continuously backlogged. In my testing, there are many .2 to .8 second intervals where higher weight queue is empty and in that duration lower weight queue get lots of job done giving the impression that there was no service differentiation.
In summary, from IO controller point of view async writes support is there. Because page cache has not been designed in such a manner that higher prio/weight writer can do more write out as compared to lower prio/weight writer, gettting service differentiation is hard and it is visible in some cases and not visible in some cases.
Do we really care that much for fairness among two writer cgroups? One can choose to do direct writes or sync writes if fairness for writes really matters for him.
Following is the only case where it is hard to ensure fairness between cgroups.
- Buffered writes Vs Buffered Writes.
So to test async writes I created two partitions on a disk and created ext3 file systems on both the partitions. Also created two cgroups and generated lots of write traffic in two cgroups (50 fio threads) and watched the disk time statistics in respective cgroups at the interval of 2 seconds. Thanks to ryo tsuruta for the test case.
*****************************************************************
sync
echo 3 >/proc/sys/vm/drop_cachesfio_args="--size=64m --rw=write --numjobs=50 --group_reporting"
echo $$ > /cgroup/bfqio/test1/tasks
fio $fio_args --name=test1 --directory=/mnt/sdd1/fio/ --output=/mnt/sdd1/fio/test1.log &echo $$ > /cgroup/bfqio/test2/tasks
fio $fio_args --name=test2 --directory=/mnt/sdd2/fio/ --output=/mnt/sdd2/fio/test2.log &
***********************************************************************And watched the disk time and sector statistics for the both the cgroups every 2 seconds using a script. How is snippet from output.
test1 statistics: time=8 48 1315 sectors=8 48 55776 dq=8 48 1
test2 statistics: time=8 48 633 sectors=8 48 14720 dq=8 48 2test1 statistics: time=8 48 5586 sectors=8 48 339064 dq=8 48 2
test2 statistics: time=8 48 2985 sectors=8 48 146656 dq=8 48 3test1 statistics: time=8 48 9935 sectors=8 48 628728 dq=8 48 3
test2 statistics: time=8 48 5265 sectors=8 48 278688 dq=8 48 4test1 statistics: time=8 48 14156 sectors=8 48 932488 dq=8 48 6
test2 statistics: time=8 48 7646 sectors=8 48 412704 dq=8 48 7test1 statistics: time=8 48 18141 sectors=8 48 1231488 dq=8 48 10
test2 statistics: time=8 48 9820 sectors=8 48 548400 dq=8 48 8test1 statistics: time=8 48 21953 sectors=8 48 1485632 dq=8 48 13
test2 statistics: time=8 48 12394 sectors=8 48 698288 dq=8 48 10test1 statistics: time=8 48 25167 sectors=8 48 1705264 dq=8 48 13
test2 statistics: time=8 48 14042 sectors=8 48 817808 dq=8 48 10First two fields in time and sectors statistics represent major and minor number of the device. Third field represents disk time in milliseconds and number of sectors transferred respectively.
So disk time consumed by group1 is almost double of group2 in this case.
Your feedback is welcome.
Thanks
VivekSignature · 2 lines
Regards Gui Jianfeng