IO scheduler based IO controller V10
Subsystems (20)
PATCH 26/28 io-controller: Per io group bdi congestion interface
In reply to Vivek Goyalo So far there used to be only one pair or queue of request descriptors (one for sync and one for async) per device and number of requests allocated used to decide whether associated bdi is congested or not.
Now with per io group request descriptor infrastructure, there is a pair of request descriptor queue per io group per device. So it might happen that overall request queue is not congested but a particular io group bio belongs to is congested.
Or, it could be otherwise that group is not congested but overall queue is congested. This can happen if user has not properly set the request descriptors limits for queue and groups.
(q->nr_requests < nr_groups * q->nr_group_requests)
Hence there is a need for new interface which can query deivce congestion status per group. This group is determined by the "struct page" IO will be done for. If page is null, then group is determined from the current task context.
o This patch introduces new set of function bdi_*_congested_group(), which take "struct page" as addition argument. These functions will call the block layer and in trun elevator to find out if the io group the page will go into is congested or not.
o Currently I have introduced the core functions and migrated most of the users. But there might be still some left. This is an ongoing TODO item.
--- block/blk-core.c | 26 ++++++++ block/blk-sysfs.c | 6 +- block/elevator-fq.c | 135 +++++++++++++++++++++++++++++++++++++++++++ block/elevator-fq.h | 24 +++++++- drivers/md/dm-table.c | 11 ++- drivers/md/dm.c | 7 +- drivers/md/dm.h | 3 +- drivers/md/linear.c | 7 ++- drivers/md/multipath.c | 7 ++- drivers/md/raid0.c | 6 +- drivers/md/raid1.c | 9 ++- drivers/md/raid10.c | 6 +- drivers/md/raid5.c | 2 +- fs/afs/write.c | 8 ++- fs/btrfs/disk-io.c | 6 +- fs/btrfs/extent_io.c | 12 ++++ fs/btrfs/volumes.c | 8 ++- fs/cifs/file.c | 11 ++++ fs/ext2/ialloc.c | 2 +- fs/gfs2/aops.c | 12 ++++ fs/nilfs2/segbuf.c | 3 +- fs/xfs/linux-2.6/xfs_aops.c | 2 +- fs/xfs/linux-2.6/xfs_buf.c | 2 +- include/linux/backing-dev.h | 63 +++++++++++++++++++- include/linux/blkdev.h | 5 ++ mm/backing-dev.c | 74 ++++++++++++++++++++++- mm/page-writeback.c | 11 ++++ mm/readahead.c | 2 +- 28 files changed, 430 insertions(+), 40 deletions(-)
block/blk-core.c +26 -0
@@ -90,6 +90,27 @@ void blk_queue_congestion_threshold(struct request_queue *q) q->nr_congestion_off = nr; } +#ifdef CONFIG_GROUP_IOSCHED +int blk_queue_io_group_congested (struct backing_dev_info *bdi,int bdi_bits, +struct page *page) +{ +int ret =0 ; +struct request_queue *q = bdi->unplug_io_data; + +if (!q || !q->elevator) +return bdi_congested (bdi, bdi_bits); + +/* Do we need to hold queue lock? */ +if (bdi_bits & (1 << BDI_sync_congested)) + ret |=elv_page_io_group_congested (q, page,1 ); + +if (bdi_bits & (1 << BDI_async_congested)) + ret |=elv_page_io_group_congested (q, page,0 ); + +return ret; +} +#endif +/** * blk_get_backing_dev_info - get the address of a queue's backing_dev_info * @bdev: device
@@ -721,6 +742,8 @@ static void __freed_request(struct request_queue *q, int sync,if (q->rq_data.count[sync] +1 <= q->nr_requests)blk_clear_queue_full (q, sync); +elv_freed_request (rl, sync); +if (rl->count[sync] +1 <= q->nr_group_requests) {if (waitqueue_active (&rl->wait[sync]))wake_up (&rl->wait[sync]);
@@ -830,6 +853,9 @@ static struct request *get_request(struct request_queue *q, int rw_flags,if (q->rq_data.count[is_sync]+1 >=queue_congestion_on_threshold (q))blk_set_queue_congested (q, is_sync); +/* check if io group will get congested after this allocation*/ +elv_get_request (rl, is_sync); +/* queue full seems redundant now */ if (q->rq_data.count[is_sync]+1 >= q->nr_requests)blk_set_queue_full (q, is_sync);
block/blk-sysfs.c +3 -3
@@ -83,9 +83,8 @@ static ssize_t queue_group_requests_show(struct request_queue *q, char *page)return queue_var_show (q->nr_group_requests, (page)); } -static ssize_t -queue_group_requests_store (struct request_queue *q,const char *page, -size_t count) +static ssize_t queue_group_requests_store (struct request_queue *q, +const char *page,size_t count) {unsigned long nr;int ret =queue_var_store (&nr, page, count);
@@ -95,6 +94,7 @@ queue_group_requests_store(struct request_queue *q, const char *page,spin_lock_irq (q->queue_lock); q->nr_group_requests = nr; +elv_updated_nr_group_requests (q);spin_unlock_irq (q->queue_lock);return ret; }
block/elevator-fq.c +135 -0
153 lines not on this page · raw message
block/elevator-fq.h +23 -1
@@ -106,6 +106,13 @@ struct io_queue { }; #ifdef CONFIG_GROUP_IOSCHED /* CONFIG_GROUP_IOSCHED */ + +enum io_group_state { + IOG_async_congested, /* The async queue of group is getting full */ + IOG_sync_congested, /* The sync queue of group is getting full */ + IOG_unused, /* Available bits start here */ +}; + struct io_group { struct io_entity entity; atomic_t ref;
@@ -141,6 +148,11 @@ struct io_group {/* Single ioq per group, used for noop, deadline, anticipatory */ struct io_queue *ioq; +/* io group congestion on and off threshold for request descriptors */ +unsigned int nr_congestion_on; +unsigned int nr_congestion_off; + +unsigned long state;/* request list associated with the group */ struct request_list rl; };
@@ -468,6 +480,11 @@ elv_get_request_list_bio(struct request_queue *q, struct bio *bio);struct request_list *elv_get_request_list_rq (struct request_queue *q,struct request *rq,int priv); +extern int elv_page_io_group_congested (struct request_queue *q, +struct page *page,int sync); +extern void elv_freed_request (struct request_list *rl,int sync); +extern void elv_get_request (struct request_list *rl,int sync); +extern void elv_updated_nr_group_requests (struct request_queue *q);#else /* !GROUP_IOSCHED */
@@ -506,9 +523,11 @@ elv_lookup_ioq_bio(struct request_queue *q, struct bio *bio) {return NULL ; } -static inline void elv_get_rl_iog (struct request_list *rl) { }static inline void elv_put_rl_iog (struct request_list *rl) { } +static inline void elv_updated_nr_group_requests (struct request_queue *q) { } +static inline void elv_freed_request (struct request_list *rl,int sync) { } +static inline void elv_get_request (struct request_list *rl,int sync) { }#endif /* GROUP_IOSCHED */
@@ -622,6 +641,9 @@ static inline struct io_queue *elv_lookup_ioq_bio(struct request_queue *q,static inline void elv_get_rl_iog (struct request_list *rl) { }static inline void elv_put_rl_iog (struct request_list *rl) { } +static inline void elv_updated_nr_group_requests (struct request_queue *q) { } +static inline void elv_freed_request (struct request_list *rl,int sync) { } +static inline void elv_get_request (struct request_list *rl,int sync) { }#endif /* CONFIG_ELV_FAIR_QUEUING */ #endif /* _ELV_SCHED_H */
drivers/md/dm-table.c +7 -4
@@ -1185,7 +1185,8 @@ int dm_table_resume_targets(struct dm_table *t)return 0 ; } -int dm_table_any_congested (struct dm_table *t,int bdi_bits) +int dm_table_any_congested (struct dm_table *t,int bdi_bits,struct page *page, +int group) {struct dm_dev_internal *dd;struct list_head *devices =dm_table_get_devices (t);
@@ -1195,9 +1196,11 @@ int dm_table_any_congested(struct dm_table *t, int bdi_bits)struct request_queue *q =bdev_get_queue (dd->dm_dev.bdev);char b[BDEVNAME_SIZE ]; -if (likely (q)) - r |=bdi_congested (&q->backing_dev_info, bdi_bits); -else +if (likely (q)) { +struct backing_dev_info *bdi = &q->backing_dev_info; + r |= group ?bdi_congested_group (bdi, bdi_bits, page) + :bdi_congested (bdi, bdi_bits); + }else DMWARN_LIMIT ("%s: any_congested: nonexistent device %s" ,dm_device_name (t->md),bdevname (dd->dm_dev.bdev, b));
drivers/md/dm.c +4 -3
@@ -1613,7 +1613,8 @@ static void dm_unplug_all(struct request_queue *q) } } -static int dm_any_congested (void *congested_data,int bdi_bits) +static int dm_any_congested (void *congested_data,int bdi_bits, +struct page *page,int group) {int r = bdi_bits;struct mapped_device *md = congested_data;
@@ -1630,8 +1631,8 @@ static int dm_any_congested(void *congested_data, int bdi_bits) r = md->queue->backing_dev_info.state & bdi_bits; else - r =dm_table_any_congested (map, bdi_bits); - + r =dm_table_any_congested (map, bdi_bits, page, + group);dm_table_put (map); } }
drivers/md/dm.h +2 -1
@@ -57,7 +57,8 @@ struct list_head *dm_table_get_devices(struct dm_table *t);void dm_table_presuspend_targets (struct dm_table *t);void dm_table_postsuspend_targets (struct dm_table *t);int dm_table_resume_targets (struct dm_table *t); -int dm_table_any_congested (struct dm_table *t,int bdi_bits); +int dm_table_any_congested (struct dm_table *t,int bdi_bits,struct page *page, +int group);int dm_table_any_busy_target (struct dm_table *t);int dm_table_set_type (struct dm_table *t);unsigned dm_table_get_type (struct dm_table *t);
drivers/md/linear.c +5 -2
@@ -102,7 +102,7 @@ static void linear_unplug(struct request_queue *q)rcu_read_unlock (); } -static int linear_congested (void *data,int bits) +static int linear_congested (void *data,int bits,struct page *page,int group) {mddev_t *mddev = data;linear_conf_t *conf;
@@ -113,7 +113,10 @@ static int linear_congested(void *data, int bits)for (i =0 ; i < mddev->raid_disks && !ret ; i++) {struct request_queue *q =bdev_get_queue (conf->disks[i].rdev->bdev); - ret |=bdi_congested (&q->backing_dev_info, bits); +struct backing_dev_info *bdi = &q->backing_dev_info; + + ret |= group ?bdi_congested_group (bdi, bits, page) : +bdi_congested (bdi, bits); }rcu_read_unlock ();
drivers/md/multipath.c +5 -2
@@ -192,7 +192,8 @@ static void multipath_status (struct seq_file *seq, mddev_t *mddev)seq_printf (seq,"]" ); } -static int multipath_congested (void *data,int bits) +static int multipath_congested (void *data,int bits,struct page *page, +int group) {mddev_t *mddev = data;multipath_conf_t *conf = mddev->private;
@@ -203,8 +204,10 @@ static int multipath_congested(void *data, int bits)mdk_rdev_t *rdev =rcu_dereference (conf->multipaths[i].rdev);if (rdev && !test_bit (Faulty, &rdev->flags)) {struct request_queue *q =bdev_get_queue (rdev->bdev); +struct backing_dev_info *bdi = &q->backing_dev_info; - ret |=bdi_congested (&q->backing_dev_info, bits); + ret |= group ?bdi_congested_group (bdi, bits, page) + :bdi_congested (bdi, bits);/* Just like multipath_map, we just check the * first available device */
drivers/md/raid0.c +4 -2
@@ -37,7 +37,7 @@ static void raid0_unplug(struct request_queue *q) } } -static int raid0_congested (void *data,int bits) +static int raid0_congested (void *data,int bits,struct page *page,int group) {mddev_t *mddev = data;raid0_conf_t *conf = mddev->private;
@@ -46,8 +46,10 @@ static int raid0_congested(void *data, int bits)for (i =0 ; i < mddev->raid_disks && !ret ; i++) {struct request_queue *q =bdev_get_queue (devlist[i]->bdev); +struct backing_dev_info *bdi = &q->backing_dev_info; - ret |=bdi_congested (&q->backing_dev_info, bits); + ret |= group ?bdi_congested_group (bdi, bits, page) + :bdi_congested (bdi, bits); }return ret; }
drivers/md/raid1.c +6 -3
@@ -570,7 +570,7 @@ static void raid1_unplug(struct request_queue *q)md_wakeup_thread (mddev->thread); } -static int raid1_congested (void *data,int bits) +static int raid1_congested (void *data,int bits,struct page *page,int group) {mddev_t *mddev = data;conf_t *conf = mddev->private;
@@ -581,14 +581,17 @@ static int raid1_congested(void *data, int bits)mdk_rdev_t *rdev =rcu_dereference (conf->mirrors[i].rdev);if (rdev && !test_bit (Faulty, &rdev->flags)) {struct request_queue *q =bdev_get_queue (rdev->bdev); +struct backing_dev_info *bdi = &q->backing_dev_info;/* Note the '|| 1' - when read_balance prefers * non-congested targets, it can be removed */ if ((bits & (1 <<BDI_async_congested)) ||1 ) - ret |=bdi_congested (&q->backing_dev_info, bits); + ret |= group ?bdi_congested_group (bdi, bits, + page) :bdi_congested (bdi, bits);else - ret &=bdi_congested (&q->backing_dev_info, bits); + ret &= group ?bdi_congested_group (bdi, bits, + page) :bdi_congested (bdi, bits); } }rcu_read_unlock ();
drivers/md/raid10.c +4 -2
@@ -625,7 +625,7 @@ static void raid10_unplug(struct request_queue *q)md_wakeup_thread (mddev->thread); } -static int raid10_congested (void *data,int bits) +static int raid10_congested (void *data,int bits,struct page *page,int group) {mddev_t *mddev = data;conf_t *conf = mddev->private;
@@ -636,8 +636,10 @@ static int raid10_congested(void *data, int bits)mdk_rdev_t *rdev =rcu_dereference (conf->mirrors[i].rdev);if (rdev && !test_bit (Faulty, &rdev->flags)) {struct request_queue *q =bdev_get_queue (rdev->bdev); +struct backing_dev_info *bdi = &q->backing_dev_info; - ret |=bdi_congested (&q->backing_dev_info, bits); + ret |= group ?bdi_congested_group (bdi, bits, page) + :bdi_congested (bdi, bits); } }rcu_read_unlock ();
drivers/md/raid5.c +1 -1
@@ -3323,7 +3323,7 @@ static void raid5_unplug_device(struct request_queue *q)unplug_slaves (mddev); } -static int raid5_congested (void *data,int bits) +static int raid5_congested (void *data,int bits,struct page *page,int group) {mddev_t *mddev = data;raid5_conf_t *conf = mddev->private;
fs/afs/write.c +7 -1
@@ -455,7 +455,7 @@ int afs_writepage(struct page *page, struct writeback_control *wbc) } wbc->nr_to_write -= ret; -if (wbc->nonblocking &&bdi_write_congested (bdi)) +if (wbc->nonblocking &&bdi_or_group_write_congested (bdi, page)) wbc->encountered_congestion =1 ;_leave (" = 0" );
@@ -491,6 +491,12 @@ static int afs_writepages_region(struct address_space *mapping,return 0 ; } +if (wbc->nonblocking &&bdi_write_congested_group (bdi, page)) { + wbc->encountered_congestion =1 ; +page_cache_release (page); +break ; + } +/* at this point we hold neither mapping->tree_lock nor lock on * the page itself: the page may be truncated or invalidated * (changing page->mapping to NULL), or even swizzled back from
fs/btrfs/disk-io.c +4 -2
@@ -1249,7 +1249,8 @@ struct btrfs_root *btrfs_read_fs_root(struct btrfs_fs_info *fs_info,return root; } -static int btrfs_congested_fn (void *congested_data,int bdi_bits) +static int btrfs_congested_fn (void *congested_data,int bdi_bits, +struct page *page,int group) {struct btrfs_fs_info *info = (struct btrfs_fs_info *)congested_data;int ret =0 ;
@@ -1260,7 +1261,8 @@ static int btrfs_congested_fn(void *congested_data, int bdi_bits)if (!device->bdev)continue ; bdi =blk_get_backing_dev_info (device->bdev); -if (bdi &&bdi_congested (bdi, bdi_bits)) { +if (bdi && (group ?bdi_congested_group (bdi, bdi_bits, page) : +bdi_congested (bdi, bdi_bits))) { ret =1 ;break ; }
fs/btrfs/extent_io.c +12 -0
@@ -2368,6 +2368,18 @@ retry:unsigned i; scanned =1 ; + +/* +* If the io group page will go into is congested, bail out. +*/ +if (wbc->nonblocking + &&bdi_write_congested_group (bdi, pvec.pages[0 ])) { + wbc->encountered_congestion =1 ; + done =1 ; +pagevec_release (&pvec); +break ; + } +for (i =0 ; i < nr_pages; i++) {struct page *page = pvec.pages[i];
fs/btrfs/volumes.c +6 -2
@@ -165,6 +165,7 @@ static noinline int run_scheduled_bios(struct btrfs_device *device)unsigned long limit;unsigned long last_waited =0 ;int force_reg =0 ; +struct page *page; bdi =blk_get_backing_dev_info (device->bdev); fs_info = device->dev_root->fs_info;
@@ -276,8 +277,11 @@ loop_lock:* is now congested. Back off and let other work structs * run instead */ -if (pending &&bdi_write_congested (bdi) && batch_run >32 && - fs_info->fs_devices->open_devices >1 ) { +if (pending) + page =bio_iovec_idx (pending,0 )->bv_page; + +if (pending &&bdi_or_group_write_congested (bdi, page) && + num_run >32 && fs_info->fs_devices->open_devices >1 ) {struct io_context *ioc; ioc = current->io_context;
fs/cifs/file.c +11 -0
@@ -1470,6 +1470,17 @@ retry: n_iov =0 ; bytes_to_write =0 ; +/* +* If the io group page will go into is congested, bail out. +*/ +if (wbc->nonblocking && +bdi_write_congested_group (bdi, pvec.pages[0 ])) { + wbc->encountered_congestion =1 ; + done =1 ; +pagevec_release (&pvec); +break ; + } +for (i =0 ; i < nr_pages; i++) { page = pvec.pages[i];/*
fs/ext2/ialloc.c +1 -1
@@ -179,7 +179,7 @@ static void ext2_preread_inode(struct inode *inode)struct backing_dev_info *bdi; bdi = inode->i_mapping->backing_dev_info; -if (bdi_read_congested (bdi)) +if (bdi_or_group_read_congested (bdi,NULL ))return ;if (bdi_write_congested (bdi))return ;
fs/gfs2/aops.c +12 -0
@@ -371,6 +371,18 @@ retry: PAGECACHE_TAG_DIRTY, min(end - index, (pgoff_t)PAGEVEC_SIZE-1) + 1))) { scanned =1 ; + +/* +* If io group page belongs to is congested. bail out. +*/ +if (wbc->nonblocking + &&bdi_write_congested_group (bdi, pvec.pages[0 ])) { + wbc->encountered_congestion =1 ; + done =1 ; +pagevec_release (&pvec); +break ; + } + ret =gfs2_write_jdata_pagevec (mapping, wbc, &pvec, nr_pages, end);if (ret) done =1 ;
fs/nilfs2/segbuf.c +2 -1
@@ -266,8 +266,9 @@ static int nilfs_submit_seg_bio(struct nilfs_write_info *wi, int mode) {struct bio *bio = wi->bio;int err; +struct page *page =bio_iovec_idx (bio,0 )->bv_page; -if (wi->nbio >0 &&bdi_write_congested (wi->bdi)) { +if (wi->nbio >0 &&bdi_or_group_write_congested (wi->bdi, page)) {wait_for_completion (&wi->bio_event); wi->nbio--;if (unlikely (atomic_read (&wi->err))) {
fs/xfs/linux-2.6/xfs_aops.c +1 -1
@@ -891,7 +891,7 @@ xfs_convert_page( bdi = inode->i_mapping->backing_dev_info; wbc->nr_to_write--; -if (bdi_write_congested (bdi)) { +if (bdi_or_group_write_congested (bdi, page)) { wbc->encountered_congestion =1 ; done =1 ; }else if (wbc->nr_to_write <=0 ) {
fs/xfs/linux-2.6/xfs_buf.c +1 -1
@@ -714,7 +714,7 @@ xfs_buf_readahead(struct backing_dev_info *bdi; bdi = target->bt_mapping->backing_dev_info; -if (bdi_read_congested (bdi)) +if (bdi_or_group_read_congested (bdi,NULL ))return ; flags |= (XBF_TRYLOCK |XBF_ASYNC |XBF_READ_AHEAD );
include/linux/backing-dev.h +60 -3
87 lines not on this page · raw message
include/linux/blkdev.h +5 -0
@@ -846,6 +846,11 @@ static inline void blk_set_queue_congested(struct request_queue *q, int sync)set_bdi_congested (&q->backing_dev_info, sync); } +#ifdef CONFIG_GROUP_IOSCHED +extern int blk_queue_io_group_congested (struct backing_dev_info *bdi, +int bdi_bits,struct page *page); +#endif +extern void blk_start_queue (struct request_queue *q);extern void blk_stop_queue (struct request_queue *q);extern void blk_sync_queue (struct request_queue *q);
mm/backing-dev.c +71 -3
96 lines not on this page · raw message
mm/page-writeback.c +11 -0
@@ -985,6 +985,17 @@ retry:if (nr_pages ==0 )break ; +/* +* If the io group page will go into is congested, bail out. +*/ +if (wbc->nonblocking + &&bdi_write_congested_group (bdi, pvec.pages[0 ])) { + wbc->encountered_congestion =1 ; + done =1 ; +pagevec_release (&pvec); +break ; + } +for (i =0 ; i < nr_pages; i++) {struct page *page = pvec.pages[i];
mm/readahead.c +1 -1
@@ -542,7 +542,7 @@ page_cache_async_readahead(struct address_space *mapping,/* * Defer asynchronous read-ahead on IO congestion. */ -if (bdi_read_congested (mapping->backing_dev_info)) +if (bdi_or_group_read_congested (mapping->backing_dev_info,NULL ))return ;/* do read-ahead */
Signature · 1 line
1.6.0.6
PATCH 27/28 io-controller: Support per cgroup per device weights and io class
In reply to Vivek GoyalThis patch enables per-cgroup per-device weight and ioprio_class handling. A new cgroup interface "policy" is introduced. You can make use of this file to configure weight and ioprio_class for each device in a given cgroup. The original "weight" and "ioprio_class" files are still available. If you don't do special configuration for a particular device, "weight" and "ioprio_class" are used as default values in this device.
You can use the following format to play with the new interface. #echo dev_major:dev_minor weight ioprio_class > /patch/to/cgroup/policy weight=0 means removing the policy for device.
Examples:
Configure weight=300 ioprio_class=2 on /dev/hdb (8:16) in this cgroup
# echo "8:16 300 2" > io.policy # cat io.policy dev weight class 8:16 300 2
Configure weight=500 ioprio_class=1 on /dev/hda (8:0) in this cgroup
# echo "8:0 500 1" > io.policy # cat io.policy dev weight class 8:0 500 1 8:16 300 2
Remove the policy for /dev/hda in this cgroup
# echo 8:0 0 1 > io.policy # cat io.policy dev weight class 8:16 300 2
Changelog (v1 -> v2)
- Rename some structures
- Use spin_lock_irqsave() and spin_lock_irqrestore() version to prevent from enabling the interrupts unconditionally.
- Fix policy setup bug when switching to another io scheduler.
- If a policy is available for a specific device, don't update weight and io class when writing "weight" and "iprio_class".
- Fix a bug when parsing policy string.
--- block/elevator-fq.c | 263 ++++++++++++++++++++++++++++++++++++++++++++++++++- block/elevator-fq.h | 10 ++ 2 files changed, 269 insertions(+), 4 deletions(-)
block/elevator-fq.c +259 -4
@@ -15,6 +15,7 @@#include <linux/blktrace_api.h> #include <linux/seq_file.h> #include <linux/biotrack.h> +#include <linux/genhd.h> #include "elevator-fq.h" const int elv_slice_sync =HZ /10 ;
@@ -1156,12 +1157,26 @@ EXPORT_SYMBOL(elv_io_group_set_async_queue);#ifdef CONFIG_GROUP_IOSCHED static void iocg_destroy (struct cgroup_subsys *subsys,struct cgroup *cgroup); -static void io_group_init_entity (struct io_cgroup *iocg,struct io_group *iog) +static struct io_policy_node *policy_search_node (const struct io_cgroup *iocg, +dev_t dev); +static void +io_group_init_entity (struct io_cgroup *iocg,struct io_group *iog,dev_t dev) {struct io_entity *entity = &iog->entity; +struct io_policy_node *pn; +unsigned long flags; + +spin_lock_irqsave (&iocg->lock, flags); + pn =policy_search_node (iocg, dev); +if (pn) { + entity->weight = pn->weight; + entity->ioprio_class = pn->ioprio_class; + }else { + entity->weight = iocg->weight; + entity->ioprio_class = iocg->ioprio_class; + } +spin_unlock_irqrestore (&iocg->lock, flags); - entity->weight = iocg->weight; - entity->ioprio_class = iocg->ioprio_class; entity->ioprio_changed =1 ; entity->my_sd = &iog->sched_data; }
@@ -1431,6 +1446,229 @@ io_cgroup_lookup_group(struct io_cgroup *iocg, void *key)return NULL ; } +static int io_cgroup_policy_read (struct cgroup *cgrp,struct cftype *cft, +struct seq_file *m) +{ +struct io_cgroup *iocg; +struct io_policy_node *pn; + + iocg =cgroup_to_io_cgroup (cgrp); + +if (list_empty (&iocg->policy_list)) + goto out; + +seq_printf (m,"dev\tweight\tclass\n" ); + +spin_lock_irq (&iocg->lock); +list_for_each_entry (pn, &iocg->policy_list, node) { +seq_printf (m,"%u:%u\t%u\t%hu\n" ,MAJOR (pn->dev), +MINOR (pn->dev), pn->weight, pn->ioprio_class); + } +spin_unlock_irq (&iocg->lock); +out: +return 0 ; +} + +static inline void policy_insert_node (struct io_cgroup *iocg, +struct io_policy_node *pn) +{ +list_add (&pn->node, &iocg->policy_list); +} + +/* Must be called with iocg->lock held */ +static inline void policy_delete_node (struct io_policy_node *pn) +{ +list_del (&pn->node); +} + +/* Must be called with iocg->lock held */ +static struct io_policy_node *policy_search_node (const struct io_cgroup *iocg, +dev_t dev) +{ +struct io_policy_node *pn; + +if (list_empty (&iocg->policy_list)) +return NULL ; + +list_for_each_entry (pn, &iocg->policy_list, node) { +if (pn->dev == dev) +return pn; + } + +return NULL ; +} + +static int check_dev_num (dev_t dev) +{ +int part =0 ; +struct gendisk *disk; + + disk =get_gendisk (dev, &part); +if (!disk || part) +return -ENODEV ; + +return 0 ; +} + +static int policy_parse_and_set (char *buf,struct io_policy_node *newpn) +{ +char *s[4 ], *p, *major_s =NULL , *minor_s =NULL ; +int ret; +unsigned long major, minor, temp; +int i =0 ; +dev_t dev; + +memset (s,0 ,sizeof (s)); +while ((p =strsep (&buf," " )) !=NULL ) { +if (!*p) +continue ; + s[i++] = p; + +/* Prevent from inputing too many things */ +if (i ==4 ) +break ; + } + +if (i !=3 ) +return -EINVAL ; + + p =strsep (&s[0 ],":" ); +if (p !=NULL ) + major_s = p; +else +return -EINVAL ; + + minor_s = s[0 ]; +if (!minor_s) +return -EINVAL ; + + ret =strict_strtoul (major_s,10 , &major); +if (ret) +return -EINVAL ; + + ret =strict_strtoul (minor_s,10 , &minor); +if (ret) +return -EINVAL ; + + dev =MKDEV (major, minor); + + ret =check_dev_num (dev); +if (ret) +return ret; + + newpn->dev = dev; + +if (s[1 ] ==NULL ) +return -EINVAL ; + + ret =strict_strtoul (s[1 ],10 , &temp); +if (ret || temp >IO_WEIGHT_MAX ) +return -EINVAL ; + + newpn->weight = temp; + +if (s[2 ] ==NULL ) +return -EINVAL ; + + ret =strict_strtoul (s[2 ],10 , &temp); +if (ret || temp <IOPRIO_CLASS_RT || temp >IOPRIO_CLASS_IDLE ) +return -EINVAL ; + newpn->ioprio_class = temp; + +return 0 ; +} + +static void update_iog_weight_prio (struct io_group *iog,struct io_cgroup *iocg, +struct io_policy_node *pn) +{ +if (pn->weight) { + iog->entity.weight = pn->weight; + iog->entity.ioprio_class = pn->ioprio_class; +/* +* iog weight and ioprio_class updating actually happens if +* ioprio_changed is set. So ensure ioprio_changed is not set +* until new weight and new ioprio_class are updated. +*/ +smp_wmb (); + iog->entity.ioprio_changed =1 ; + }else { + iog->entity.weight = iocg->weight; + iog->entity.ioprio_class = iocg->ioprio_class; + +/* The same as above */ +smp_wmb (); + iog->entity.ioprio_changed =1 ; + } +} + +static int io_cgroup_policy_write (struct cgroup *cgrp,struct cftype *cft, +const char *buffer) +{ +struct io_cgroup *iocg; +struct io_policy_node *newpn, *pn; +char *buf; +int ret =0 ; +int keep_newpn =0 ; +struct hlist_node *n; +struct io_group *iog; + + buf =kstrdup (buffer,GFP_KERNEL ); +if (!buf) +return -ENOMEM ; + + newpn =kzalloc (sizeof (*newpn),GFP_KERNEL ); +if (!newpn) { + ret = -ENOMEM ; + goto free_buf; + } + + ret =policy_parse_and_set (buf, newpn); +if (ret) + goto free_newpn; + +if (!cgroup_lock_live_group (cgrp)) { + ret = -ENODEV ; + goto free_newpn; + } + + iocg =cgroup_to_io_cgroup (cgrp); +spin_lock_irq (&iocg->lock); + + pn =policy_search_node (iocg, newpn->dev); +if (!pn) { +if (newpn->weight !=0 ) { +policy_insert_node (iocg, newpn); + keep_newpn =1 ; + } + goto update_io_group; + } + +if (newpn->weight ==0 ) { +/* weight == 0 means deleteing a policy */ +policy_delete_node (pn); + goto update_io_group; + } + + pn->weight = newpn->weight; + pn->ioprio_class = newpn->ioprio_class; + +update_io_group: +hlist_for_each_entry (iog, n, &iocg->group_data, group_node) { +if (iog->dev == newpn->dev) +update_iog_weight_prio (iog, iocg, newpn); + } +spin_unlock_irq (&iocg->lock); + +cgroup_unlock (); + +free_newpn: +if (!keep_newpn) +kfree (newpn); +free_buf: +kfree (buf); +return ret; +} +#define SHOW_FUNCTION (__VAR) \ static u64 io_cgroup_##__VAR##_read(struct cgroup *cgroup, \ struct cftype *cftype) \
@@ -1463,6 +1701,7 @@ static int io_cgroup_##__VAR##_write(struct cgroup *cgroup, \struct io_cgroup *iocg; \struct io_group *iog; \struct hlist_node *n; \ +struct io_policy_node *pn; \ \if (val < (__MIN) || val > (__MAX)) \return -EINVAL ; \
@@ -1475,6 +1714,9 @@ static int io_cgroup_##__VAR##_write(struct cgroup *cgroup, \spin_lock_irq (&iocg->lock); \ iocg->__VAR = (unsigned long )val; \ hlist_for_each_entry(iog, n, &iocg->group_data, group_node) { \ + pn =policy_search_node (iocg, iog->dev); \ +if (pn) \ +continue ; \ iog->entity.__VAR = (unsigned long )val; \smp_wmb (); \ iog->entity.ioprio_changed =1 ; \
@@ -1610,6 +1852,12 @@ static int io_cgroup_disk_dequeue_read(struct cgroup *cgroup,struct cftype io_files[] = { { + .name ="policy" , + .read_seq_string = io_cgroup_policy_read, + .write_string = io_cgroup_policy_write, + .max_write_len =256 , + }, + { .name ="weight" , .read_u64 = io_cgroup_weight_read, .write_u64 = io_cgroup_weight_write,
@@ -1660,6 +1908,7 @@ static struct cgroup_subsys_state *iocg_create(struct cgroup_subsys *subsys,INIT_HLIST_HEAD (&iocg->group_data); iocg->weight =IO_WEIGHT_DEFAULT ; iocg->ioprio_class =IOPRIO_CLASS_BE ; +INIT_LIST_HEAD (&iocg->policy_list);return &iocg->css; }
@@ -1753,7 +2002,7 @@ io_group_chain_alloc(struct request_queue *q, void *key, struct cgroup *cgroup)sscanf (dev_name (bdi->dev),"%u:%u" , &major, &minor); iog->dev =MKDEV (major, minor); -io_group_init_entity (iocg, iog); +io_group_init_entity (iocg, iog, iog->dev);atomic_set (&iog->ref,0 );
@@ -2109,6 +2358,7 @@ static void iocg_destroy(struct cgroup_subsys *subsys, struct cgroup *cgroup)struct io_group *iog;struct elv_fq_data *efqd;unsigned long uninitialized_var (flags ); +struct io_policy_node *pn, *pntmp;/* * io groups are linked in two lists. One list is maintained
@@ -2148,6 +2398,11 @@ remove_entry: goto remove_entry; done: +list_for_each_entry_safe (pn, pntmp, &iocg->policy_list, node) { +policy_delete_node (pn); +kfree (pn); + } +free_css_id (&io_subsys, &iocg->css);rcu_read_unlock ();BUG_ON (!hlist_empty (&iocg->group_data));
block/elevator-fq.h +10 -0
@@ -157,12 +157,22 @@ struct io_group {struct request_list rl; }; +struct io_policy_node { +struct list_head node; +dev_t dev; +unsigned int weight; +unsigned short ioprio_class; +}; +struct io_cgroup {struct cgroup_subsys_state css;unsigned int weight;unsigned short ioprio_class; +/* list of io_policy_node */ +struct list_head policy_list; +spinlock_t lock;struct hlist_head group_data; };
Signature · 1 line
1.6.0.6
PATCH 28/28 io-controller: debug elevator fair queuing support
In reply to Vivek Goyalo More debugging help to debug elevator fair queuing support. Enabled under CONFIG_DEBUG_ELV_FAIR_QUEUING. Currently it prints vdisktime related trace messages in blktrace.
--- block/Kconfig.iosched | 9 +++++++++ block/elevator-fq.c | 47 ++++++++++++++++++++++++++++++++++++++++++++++- 2 files changed, 55 insertions(+), 1 deletions(-)
block/Kconfig.iosched +9 -0
@@ -15,6 +15,15 @@ config ELV_FAIR_QUEUING other ioschedulers can make use of it. If unsure, say N. +config DEBUG_ELV_FAIR_QUEUING + bool "Debug elevator fair queuing" + depends on ELV_FAIR_QUEUING + default n + ---help--- + Enable some debugging hooks for elevator fair queuing support. + Currently it just outputs more information about vdisktime in + blktrace output . + config IOSCHED_NOOP bool default y
block/elevator-fq.c +46 -1
74 lines not on this page · raw message
Signature · 1 line
1.6.0.6
Andrew Morton
maintainerQuoted 5 lines · On Thu, 24 Sep 2009 15:25:04 -0400 Vivek Goyal <[email protected]> wrote:
Hi All,
Here is the V10 of the IO controller patches generated on top of 2.6.31.
Thanks for the writeup. It really helps and is most worthwhile for a project of this importance, size and complexity.
Quoted 76 lines
What problem are we trying to solve
===================================
Provide group IO scheduling feature in Linux along the lines of other resource controllers like cpu.IOW, provide facility so that a user can group applications using cgroups and control the amount of disk time/bandwidth received by a group based on its weight.
How to solve the problem
=========================Different people have solved the issue differetnly. So far looks it looks like we seem to have following two core requirements when it comes to fairness at group level.
- Control bandwidth seen by groups.
- Control on latencies when a request gets backlogged in group.At least there are now three patchsets available (including this one).
IO throttling
-------------
This is a bandwidth controller which keeps track of IO rate of a group and throttles the process in the group if it exceeds the user specified limit.dm-ioband
---------
This is a proportional bandwidth controller implemented as device mapper driver and provides fair access in terms of amount of IO done (not in terms of disk time as CFQ does).So one will setup one or more dm-ioband devices on top of physical/logical block device, configure the ioband device and pass information like grouping etc. Now this device will keep track of bios flowing through it and control the flow of bios based on group policies.
IO scheduler based IO controller
--------------------------------
Here we have viewed the problem of IO contoller as hierarchical group scheduling (along the lines of CFS group scheduling) issue. Currently one can view linux IO schedulers as flat where there is one root group and all the IO belongs to that group.This patchset basically modifies IO schedulers to also support hierarchical group scheduling. CFQ already provides fairness among different processes. I have extended it support group IO schduling. Also took some of the code out of CFQ and put in a common layer so that same group scheduling code can be used by noop, deadline and AS to support group scheduling.
Pros/Cons
=========
There are pros and cons to each of the approach. Following are some of the thoughts.Max bandwidth vs proportional bandwidth
---------------------------------------
IO throttling is a max bandwidth controller and not a proportional one. Additionaly it provides fairness in terms of amount of IO done (and not in terms of disk time as CFQ does).Personally, I think that proportional weight controller is useful to more people than just max bandwidth controller. In addition, IO scheduler based controller can also be enhanced to do max bandwidth control. So it can satisfy wider set of requirements.
Fairness in terms of disk time vs size of IO
---------------------------------------------
An higher level controller will most likely be limited to providing fairness in terms of size/number of IO done and will find it hard to provide fairness in terms of disk time used (as CFQ provides between various prio levels). This is because only IO scheduler knows how much disk time a queue has used and information about queues and disk time used is not exported to higher layers.
Quoted 2 lines
So a seeky application will still run away with lot of disk time and bring down the overall throughput of the the disk.
But that's only true if the thing is poorly implemented.
A high-level controller will need some view of the busyness of the underlying device(s). That could be "proportion of idle time", or "average length of queue" or "average request latency" or some mix of these or something else altogether.
But these things are simple to calculate, and are simple to feed back to the higher-level controller and probably don't require any changes to to IO scheduler at all, which is a great advantage.
And I must say that high-level throttling based upon feedback from lower layers seems like a much better model to me than hacking away in the IO scheduler layer. Both from an implementation point of view and from a "we can get it to work on things other than block devices" point of view.
Quoted 8 lines
Currently dm-ioband provides fairness in terms of number/size of IO.
Latencies and isolation between groups
--------------------------------------
An higher level controller is generally implementing a bandwidth throttling solution where if a group exceeds either the max bandwidth or the proportional share then throttle that group.
Quoted 6 lines
This kind of approach will probably not help in controlling latencies as it will depend on underlying IO scheduler. Consider following scenario.
Assume there are two groups. One group is running multiple sequential readers and other group has a random reader. sequential readers will get a nice 100ms slice
Do you refer to each reader within group1, or to all readers? It would be daft if each reader in group1 were to get 100ms.
Quoted 3 lines
each and then a random reader from group2 will get to dispatch the request. So latency of this random reader will depend on how many sequential readers are running in other group and that is a weak isolation between groups.
And yet that is what you appear to mean.
But surely nobody would do that - the 100ms would be assigned to and distributed amongst all readers in group1?
Quoted 6 lines
When we control things at IO scheduler level, we assign one time slice to one group and then pick next entity to run. So effectively after one time slice (max 180ms, if prio 0 sequential reader is running), random reader in other group will get to run. Hence we achieve better isolation between groups as response time of process in a differnt group is generally not dependent on number of processes running in competing group.
I don't understand why you're comparing this implementation with such an obviously dumb competing design!
Quoted 17 lines
So a higher level solution is most likely limited to only shaping bandwidth without any control on latencies.
Stacking group scheduler on top of CFQ can lead to issues
---------------------------------------------------------
IO throttling and dm-ioband both are second level controller. That is these controllers are implemented in higher layers than io schedulers. So they control the IO at higher layer based on group policies and later IO schedulers take care of dispatching these bios to disk.Implementing a second level controller has the advantage of being able to provide bandwidth control even on logical block devices in the IO stack which don't have any IO schedulers attached to these. But they can also interefere with IO scheduling policy of underlying IO scheduler and change the effective behavior. Following are some of the issues which I think should be visible in second level controller in one form or other.
Quoted 7 lines
Prio with-in group
------------------
A second level controller can potentially interefere with behavior of different prio processes with-in a group. bios are buffered at higher layer in single queue and release of bios is FIFO and not proportionate to the ioprio of the process. This can result in a particular prio level not getting fair share.
That's an administrator error, isn't it? Should have put the different-priority processes into different groups.
Quoted 5 lines
Buffering at higher layer can delay read requests for more than slice idle period of CFQ (default 8 ms). That means, it is possible that we are waiting for a request from the queue but it is buffered at higher layer and then idle timer will fire. It means that queue will losse its share at the same time overall throughput will be impacted as we lost those 8 ms.
That sounds like a bug.
Quoted 10 lines
Read Vs Write
-------------
Writes can overwhelm readers hence second level controller FIFO release will run into issue here. If there is a single queue maintained then reads will suffer large latencies. If there separate queues for reads and writes then it will be hard to decide in what ratio to dispatch reads and writes as it is IO scheduler's decision to decide when and how much read/write to dispatch. This is another place where higher level controller will not be in sync with lower level io scheduler and can change the effective policies of underlying io scheduler.
The IO schedulers already take care of read-vs-write and already take care of preventing large writes-starve-reads latencies (or at least, they're supposed to).
Quoted 4 lines
CFQ IO context Issues
---------------------
Buffering at higher layer means submission of bios later with the help of a worker thread.
Why?
If it's a read, we just block the userspace process.
If it's a delayed write, the IO submission already happens in a kernel thread.
If it's a synchronous write, we have to block the userspace caller anyway.
Async reads might be an issue, dunno.
Quoted 4 lines
This changes the io context information at CFQ layer which
assigns the request to submitting thread. Change of io context info again leads to issues of idle timer expiry and issue of a process not getting fair share and reduced throughput.
But we already have that problem with delayed writeback, which is a huge thing - often it's the majority of IO.
Quoted 14 lines
Throughput with noop, deadline and AS
---------------------------------------------
I think an higher level controller will result in reduced overall throughput (as compared to io scheduler based io controller) and more seeks with noop, deadline and AS.The reason being, that it is likely that IO with-in a group will be related and will be relatively close as compared to IO across the groups. For example, thread pool of kvm-qemu doing IO for virtual machine. In case of higher level control, IO from various groups will go into a single queue at lower level controller and it might happen that IO is now interleaved (G1, G2, G1, G3, G4....) causing more seeks and reduced throughput. (Agreed that merging will help up to some extent but still....).
Quoted 7 lines
Instead, in case of lower level controller, IO scheduler maintains one queue per group hence there is no interleaving of IO between groups. And if IO is related with-in group, then we shoud get reduced number/amount of seek and higher throughput.
Latency can be a concern but that can be controlled by reducing the time slice length of the queue.
Well maybe, maybe not. If a group is throttled, it isn't submitting new IO. The unthrottled group is doing the IO submitting and that IO will have decent locality.
Quoted 30 lines
Fairness at logical device level vs at physical device level
------------------------------------------------------------IO scheduler based controller has the limitation that it works only with the bottom most devices in the IO stack where IO scheduler is attached.
For example, assume a user has created a logical device lv0 using three underlying disks sda, sdb and sdc. Also assume there are two tasks T1 and T2 in two groups doing IO on lv0. Also assume that weights of groups are in the ratio of 2:1 so T1 should get double the BW of T2 on lv0 device.
T1 T2 \ / lv0 / | \ sda sdb sdcNow resource control will take place only on devices sda, sdb and sdc and not at lv0 level. So if IO from two tasks is relatively uniformly distributed across the disks then T1 and T2 will see the throughput ratio in proportion to weight specified. But if IO from T1 and T2 is going to different disks and there is no contention then at higher level they both will see same BW.
Here a second level controller can produce better fairness numbers at logical device but most likely at redued overall throughput of the system, because it will try to control IO even if there is no contention at phsical possibly leaving diksks unused in the system.
Quoted 5 lines
Hence, question comes that how important it is to control bandwidth at higher level logical devices also. The actual contention for resources is at the leaf block device so it probably makes sense to do any kind of control there and not at the intermediate devices. Secondly probably it also means better use of available resources.
hm. What will be the effects of this limitation in real-world use?
Quoted 9 lines
Limited Fairness
----------------
Currently CFQ idles on a sequential reader queue to make sure it gets its fair share. A second level controller will find it tricky to anticipate. Either it will not have any anticipation logic and in that case it will not provide fairness to single readers in a group (as dm-ioband does) or if it starts anticipating then we should run into these strange situations where second level controller is anticipating on one queue/group and underlying IO scheduler might be anticipating on something else.
It depends on the size of the inter-group timeslices. If the amount of time for which a group is unthrottled is "large" comapred to the typical anticipation times, this issue fades away.
And those timeslices _should_ be large. Because as you mentioned above, different groups are probably working different parts of the disk.
Quoted 22 lines
Need of device mapper tools
---------------------------
A device mapper based solution will require creation of a ioband device on each physical/logical device one wants to control. So it requires usage of device mapper tools even for the people who are not using device mapper. At the same time creation of ioband device on each partition in the system to control the IO can be cumbersome and overwhelming if system has got lots of disks and partitions with-in.IMHO, IO scheduler based IO controller is a reasonable approach to solve the problem of group bandwidth control, and can do hierarchical IO scheduling more tightly and efficiently.
But I am all ears to alternative approaches and suggestions how doing things can be done better and will be glad to implement it.
TODO
====
- code cleanups, testing, bug fixing, optimizations, benchmarking etc...
- More testing to make sure there are no regressions in CFQ.
Quoted 6 lines
Testing
=======Environment
==========
A 7200 RPM SATA drive with queue depth of 31. Ext3 filesystem.
That's a bit of a toy.
Do we have testing results for more enterprisey hardware? Big storage arrays? SSD? Infiniband? iscsi? nfs? (lol, gotcha)
Quoted 27 lines
I am mostly
running fio jobs which have been limited to 30 seconds run and then monitored the throughput and latency.Test1: Random Reader Vs Random Writers
======================================
Launched a random reader and then increasing number of random writers to see the effect on random reader BW and max lantecies.[fio --rw=randwrite --bs=64K --size=2G --runtime=30 --direct=1 --ioengine=libaio --iodepth=4 --numjobs= <1 to 32> ] [fio --rw=randread --bs=4K --size=2G --runtime=30 --direct=1]
[Vanilla CFQ, No groups]
<--------------random writers--------------------> <------random reader-->nr Max-bdwidth Min-bdwidth Agg-bdwidth Max-latency Agg-bdwidth Max-latency 1 5737KiB/s 5737KiB/s 5737KiB/s 164K usec 503KiB/s 159K usec 2 2055KiB/s 1984KiB/s 4039KiB/s 1459K usec 150KiB/s 170K usec 4 1238KiB/s 932KiB/s 4419KiB/s 4332K usec 153KiB/s 225K usec 8 1059KiB/s 929KiB/s 7901KiB/s 1260K usec 118KiB/s 377K usec 16 604KiB/s 483KiB/s 8519KiB/s 3081K usec 47KiB/s 756K usec 32 367KiB/s 222KiB/s 9643KiB/s 5940K usec 22KiB/s 923K usecCreated two cgroups group1 and group2 of weights 500 each. Launched increasing number of random writers in group1 and one random reader in group2 using fio.
[IO controller CFQ; group_idle=8; group1 weight=500; group2 weight=500] <--------------random writers(group1)-------------> <-random reader(group2)->
Quoted 7 lines
nr Max-bdwidth Min-bdwidth Agg-bdwidth Max-latency Agg-bdwidth Max-latency 1 18115KiB/s 18115KiB/s 18115KiB/s 604K usec 345KiB/s 176K usec 2 3752KiB/s 3676KiB/s 7427KiB/s 4367K usec 402KiB/s 187K usec 4 1951KiB/s 1863KiB/s 7642KiB/s 1989K usec 384KiB/s 181K usec 8 755KiB/s 629KiB/s 5683KiB/s 2133K usec 366KiB/s 319K usec 16 418KiB/s 369KiB/s 6276KiB/s 1323K usec 352KiB/s 287K usec 32 236KiB/s 191KiB/s 6518KiB/s 1910K usec 337KiB/s 273K usec
That's a good result.
Quoted 13 lines
Also ran the same test with IO controller CFQ in flat mode to see if there are any major deviations from Vanilla CFQ. Does not look like any.
[IO controller CFQ; No groups ]
<--------------random writers--------------------> <------random reader-->nr Max-bdwidth Min-bdwidth Agg-bdwidth Max-latency Agg-bdwidth Max-latency 1 5696KiB/s 5696KiB/s 5696KiB/s 259K usec 500KiB/s 194K usec 2 2483KiB/s 2197KiB/s 4680KiB/s 887K usec 150KiB/s 159K usec 4 1471KiB/s 1433KiB/s 5817KiB/s 962K usec 126KiB/s 189K usec 8 691KiB/s 580KiB/s 5159KiB/s 2752K usec 197KiB/s 246K usec 16 781KiB/s 698KiB/s 11892KiB/s 943K usec 61KiB/s 529K usec 32 415KiB/s 324KiB/s 12461KiB/s 4614K usec 17KiB/s 737K usec
Quoted 3 lines
Notes:
- With vanilla CFQ, random writers can overwhelm a random reader. Bring down its throughput and bump up latencies significantly.
Isn't that a CFQ shortcoming which we should address separately? If so, the comparisons aren't presently valid because we're comparing with a CFQ which has known, should-be-fixed problems.
Quoted 109 lines
- With IO controller, one can provide isolation to the random reader group and maintain consitent view of bandwidth and latencies.
Test2: Random Reader Vs Sequential Reader
========================================
Launched a random reader and then increasing number of sequential readers to see the effect on BW and latencies of random reader.[fio --rw=read --bs=4K --size=2G --runtime=30 --direct=1 --numjobs= <1 to 16> ] [fio --rw=randread --bs=4K --size=2G --runtime=30 --direct=1]
[ Vanilla CFQ, No groups ]
<---------------seq readers----------------------> <------random reader-->nr Max-bdwidth Min-bdwidth Agg-bdwidth Max-latency Agg-bdwidth Max-latency 1 23318KiB/s 23318KiB/s 23318KiB/s 55940 usec 36KiB/s 247K usec 2 14732KiB/s 11406KiB/s 26126KiB/s 142K usec 20KiB/s 446K usec 4 9417KiB/s 5169KiB/s 27338KiB/s 404K usec 10KiB/s 993K usec 8 3360KiB/s 3041KiB/s 25850KiB/s 954K usec 60KiB/s 956K usec 16 1888KiB/s 1457KiB/s 26763KiB/s 1871K usec 28KiB/s 1868K usecCreated two cgroups group1 and group2 of weights 500 each. Launched increasing number of sequential readers in group1 and one random reader in group2 using fio.
[IO controller CFQ; group_idle=1; group1 weight=500; group2 weight=500] <---------------group1---------------------------> <------group2--------->
nr Max-bdwidth Min-bdwidth Agg-bdwidth Max-latency Agg-bdwidth Max-latency 1 13733KiB/s 13733KiB/s 13733KiB/s 247K usec 330KiB/s 154K usec 2 8553KiB/s 4963KiB/s 13514KiB/s 472K usec 322KiB/s 174K usec 4 5045KiB/s 1367KiB/s 13134KiB/s 947K usec 318KiB/s 178K usec 8 1774KiB/s 1420KiB/s 13035KiB/s 1871K usec 323KiB/s 233K usec 16 959KiB/s 518KiB/s 12691KiB/s 3809K usec 324KiB/s 208K usecAlso ran the same test with IO controller CFQ in flat mode to see if there are any major deviations from Vanilla CFQ. Does not look like any.
[IO controller CFQ; No groups ]
<---------------seq readers----------------------> <------random reader-->nr Max-bdwidth Min-bdwidth Agg-bdwidth Max-latency Agg-bdwidth Max-latency 1 23028KiB/s 23028KiB/s 23028KiB/s 47460 usec 36KiB/s 253K usec 2 14452KiB/s 11176KiB/s 25628KiB/s 145K usec 20KiB/s 447K usec 4 8815KiB/s 5720KiB/s 27121KiB/s 396K usec 10KiB/s 968K usec 8 3335KiB/s 2827KiB/s 24866KiB/s 960K usec 62KiB/s 955K usec 16 1784KiB/s 1311KiB/s 26537KiB/s 1883K usec 26KiB/s 1866K usecNotes:
- The BW and latencies of random reader in group 2 seems to be stable and bounded and does not get impacted much as number of sequential readers increase in group1. Hence provding good isolation.- Throughput of sequential readers comes down and latencies go up as half of disk bandwidth (in terms of time) has been reserved for random reader group.
Test3: Sequential Reader Vs Sequential Reader
============================================
Created two cgroups group1 and group2 of weights 500 and 1000 respectively. Launched increasing number of sequential readers in group1 and one sequential reader in group2 using fio and monitored how bandwidth is being distributed between two groups.First 5 columns give stats about job in group1 and last two columns give stats about job in group2.
<---------------group1---------------------------> <------group2--------->
nr Max-bdwidth Min-bdwidth Agg-bdwidth Max-latency Agg-bdwidth Max-latency 1 8970KiB/s 8970KiB/s 8970KiB/s 230K usec 20681KiB/s 124K usec 2 6783KiB/s 3202KiB/s 9984KiB/s 546K usec 19682KiB/s 139K usec 4 4641KiB/s 1029KiB/s 9280KiB/s 1185K usec 19235KiB/s 172K usec 8 1435KiB/s 1079KiB/s 9926KiB/s 2461K usec 19501KiB/s 153K usec 16 764KiB/s 398KiB/s 9395KiB/s 4986K usec 19367KiB/s 172K usecNote: group2 is getting double the bandwidth of group1 even in the face of increasing number of readers in group1.
Test4 (Isolation between two KVM virtual machines)
==================================================
Created two KVM virtual machines. Partitioned a disk on host in two partitions and gave one partition to each virtual machine. Put both the virtual machines in two different cgroup of weight 1000 and 500 each. Virtual machines created ext3 file system on the partitions exported from host and did buffered writes. Host seems writes as synchronous and virtual machine with higher weight gets double the disk time of virtual machine of lower weight. Used deadline scheduler in this test case.Some more details about configuration are in documentation patch.
Test5 (Fairness for async writes, Buffered Write Vs Buffered Write)
===================================================================
Fairness for async writes is tricky and biggest reason is that async writes are cached in higher layers (page cahe) as well as possibly in file system layer also (btrfs, xfs etc), and are dispatched to lower layers not necessarily in proportional manner.For example, consider two dd threads reading
/dev/zeroas input file and doing writes of huge files. Very soon we will crossvm_dirty_ratioand dd thread will be forced to write out some pages to disk before more pages can be dirtied. But not necessarily dirty pages of same thread are picked. It can very well pick the inode of lesser priority dd thread and do some writeout. So effectively higher weight dd is doing writeouts of lower weight dd pages and we don't see service differentation.IOW, the core problem with buffered write fairness is that higher weight thread does not throw enought IO traffic at IO controller to keep the queue continuously backlogged. In my testing, there are many .2 to .8 second intervals where higher weight queue is empty and in that duration lower weight queue get lots of job done giving the impression that there was no service differentiation.
Quoted 5 lines
In summary, from IO controller point of view async writes support is there. Because page cache has not been designed in such a manner that higher prio/weight writer can do more write out as compared to lower prio/weight writer, gettting service differentiation is hard and it is visible in some cases and not visible in some cases.
Here's where it all falls to pieces.
For async writeback we just don't care about IO priorities. Because from the point of view of the userspace task, the write was async! It occurred at memory bandwidth speed.
It's only when the kernel's dirty memory thresholds start to get exceeded that we start to care about prioritisation. And at that time, all dirty memory (within a memcg?) is equal - a high-ioprio dirty page consumes just as much memory as a low-ioprio dirty page.
So when balance_dirty_pages() hits, what do we want to do?
I suppose that all we can do is to block low-ioprio processes more agressively at the VFS layer, to reduce the rate at which they're dirtying memory so as to give high-ioprio processes more of the disk bandwidth.
But you've gone and implemented all of this stuff at the io-controller level and not at the VFS level so you're, umm, screwed.
Importantly screwed! It's a very common workload pattern, and one which causes tremendous amounts of IO to be generated very quickly, traditionally causing bad latency effects all over the place. And we have no answer to this.
Quoted 113 lines
Vanilla CFQ Vs IO Controller CFQ
================================
We have not fundamentally changed CFQ, instead enhanced it to also support hierarchical io scheduling. In the process invariably there are small changes here and there as new scenarios come up. Running some tests here and comparing both the CFQ's to see if there is any major deviation in behavior.Test1: Sequential Readers
=========================
[fio --rw=read --bs=4K --size=2G --runtime=30 --direct=1 --numjobs=<1 to 16> ]IO scheduler: Vanilla CFQ
nr Max-bdwidth Min-bdwidth Agg-bdwidth Max-latency 1 35499KiB/s 35499KiB/s 35499KiB/s 19195 usec 2 17089KiB/s 13600KiB/s 30690KiB/s 118K usec 4 9165KiB/s 5421KiB/s 29411KiB/s 380K usec 8 3815KiB/s 3423KiB/s 29312KiB/s 830K usec 16 1911KiB/s 1554KiB/s 28921KiB/s 1756K usecIO scheduler: IO controller CFQ
nr Max-bdwidth Min-bdwidth Agg-bdwidth Max-latency 1 34494KiB/s 34494KiB/s 34494KiB/s 14482 usec 2 16983KiB/s 13632KiB/s 30616KiB/s 123K usec 4 9237KiB/s 5809KiB/s 29631KiB/s 372K usec 8 3901KiB/s 3505KiB/s 29162KiB/s 822K usec 16 1895KiB/s 1653KiB/s 28945KiB/s 1778K usecTest2: Sequential Writers
=========================
[fio --rw=write --bs=4K --size=2G --runtime=30 --direct=1 --numjobs=<1 to 16> ]IO scheduler: Vanilla CFQ
nr Max-bdwidth Min-bdwidth Agg-bdwidth Max-latency 1 22669KiB/s 22669KiB/s 22669KiB/s 401K usec 2 14760KiB/s 7419KiB/s 22179KiB/s 571K usec 4 5862KiB/s 5746KiB/s 23174KiB/s 444K usec 8 3377KiB/s 2199KiB/s 22427KiB/s 1057K usec 16 2229KiB/s 556KiB/s 20601KiB/s 5099K usecIO scheduler: IO Controller CFQ
nr Max-bdwidth Min-bdwidth Agg-bdwidth Max-latency 1 22911KiB/s 22911KiB/s 22911KiB/s 37319 usec 2 11752KiB/s 11632KiB/s 23383KiB/s 245K usec 4 6663KiB/s 5409KiB/s 23207KiB/s 384K usec 8 3161KiB/s 2460KiB/s 22566KiB/s 935K usec 16 1888KiB/s 795KiB/s 21349KiB/s 3009K usecTest3: Random Readers
=========================
[fio --rw=randread --bs=4K --size=2G --runtime=30 --direct=1 --numjobs=1 to 16]IO scheduler: Vanilla CFQ
nr Max-bdwidth Min-bdwidth Agg-bdwidth Max-latency 1 484KiB/s 484KiB/s 484KiB/s 22596 usec 2 229KiB/s 196KiB/s 425KiB/s 51111 usec 4 119KiB/s 73KiB/s 405KiB/s 2344 msec 8 93KiB/s 23KiB/s 399KiB/s 2246 msec 16 38KiB/s 8KiB/s 328KiB/s 3965 msecIO scheduler: IO Controller CFQ
nr Max-bdwidth Min-bdwidth Agg-bdwidth Max-latency 1 483KiB/s 483KiB/s 483KiB/s 29391 usec 2 229KiB/s 196KiB/s 426KiB/s 51625 usec 4 132KiB/s 88KiB/s 417KiB/s 2313 msec 8 79KiB/s 18KiB/s 389KiB/s 2298 msec 16 43KiB/s 9KiB/s 327KiB/s 3905 msecTest4: Random Writers
=====================
[fio --rw=randwrite --bs=4K --size=2G --runtime=30 --direct=1 --numjobs=1 to 16]IO scheduler: Vanilla CFQ
nr Max-bdwidth Min-bdwidth Agg-bdwidth Max-latency 1 14641KiB/s 14641KiB/s 14641KiB/s 93045 usec 2 7896KiB/s 1348KiB/s 9245KiB/s 82778 usec 4 2657KiB/s 265KiB/s 6025KiB/s 216K usec 8 951KiB/s 122KiB/s 3386KiB/s 1148K usec 16 66KiB/s 22KiB/s 829KiB/s 1308 msecIO scheduler: IO Controller CFQ
nr Max-bdwidth Min-bdwidth Agg-bdwidth Max-latency 1 14454KiB/s 14454KiB/s 14454KiB/s 74623 usec 2 4595KiB/s 4104KiB/s 8699KiB/s 135K usec 4 3113KiB/s 334KiB/s 5782KiB/s 200K usec 8 1146KiB/s 95KiB/s 3832KiB/s 593K usec 16 71KiB/s 29KiB/s 814KiB/s 1457 msecNotes:
- Does not look like that anything has changed significantly.Previous versions of the patches were posted here.
------------------------------------------------(V1) http://lkml.org/lkml/2009/3/11/486
(V2) http://lkml.org/lkml/2009/5/5/275
(V3) http://lkml.org/lkml/2009/5/26/472
(V4) http://lkml.org/lkml/2009/6/8/580
(V5) http://lkml.org/lkml/2009/6/19/279
(V6) http://lkml.org/lkml/2009/7/2/369
(V7) http://lkml.org/lkml/2009/7/24/253
(V8) http://lkml.org/lkml/2009/8/16/204
(V9) http://lkml.org/lkml/2009/8/28/327Thanks
Vivek
Quoted 41 lines · On Thu, 24 Sep 2009 14:33:15 -0700 Andrew Morton <[email protected]>…
Quoted 22 lines
Test5 (Fairness for async writes, Buffered Write Vs Buffered Write)
===================================================================
Fairness for async writes is tricky and biggest reason is that async writes are cached in higher layers (page cahe) as well as possibly in file system layer also (btrfs, xfs etc), and are dispatched to lower layers not necessarily in proportional manner.For example, consider two dd threads reading
/dev/zeroas input file and doing writes of huge files. Very soon we will crossvm_dirty_ratioand dd thread will be forced to write out some pages to disk before more pages can be dirtied. But not necessarily dirty pages of same thread are picked. It can very well pick the inode of lesser priority dd thread and do some writeout. So effectively higher weight dd is doing writeouts of lower weight dd pages and we don't see service differentation.IOW, the core problem with buffered write fairness is that higher weight thread does not throw enought IO traffic at IO controller to keep the queue continuously backlogged. In my testing, there are many .2 to .8 second intervals where higher weight queue is empty and in that duration lower weight queue get lots of job done giving the impression that there was no service differentiation.
Quoted 5 lines
In summary, from IO controller point of view async writes support is there. Because page cache has not been designed in such a manner that higher prio/weight writer can do more write out as compared to lower prio/weight writer, gettting service differentiation is hard and it is visible in some cases and not visible in some cases.
Here's where it all falls to pieces.
For async writeback we just don't care about IO priorities. Because from the point of view of the userspace task, the write was async! It occurred at memory bandwidth speed.
It's only when the kernel's dirty memory thresholds start to get exceeded that we start to care about prioritisation. And at that time, all dirty memory (within a memcg?) is equal - a high-ioprio dirty page consumes just as much memory as a low-ioprio dirty page.
So when
balance_dirty_pages()hits, what do we want to do?
Quoted 8 lines
I suppose that all we can do is to block low-ioprio processes more agressively at the VFS layer, to reduce the rate at which they're dirtying memory so as to give high-ioprio processes more of the disk bandwidth.
But you've gone and implemented all of this stuff at the io-controller level and not at the VFS level so you're, umm, screwed.
I think I must support dirty-ratio in memcg layer. But not yet. I can't easily imagine how the system will work if both dirty-ratio and io-controller cgroup are supported. But considering use them as a set of cgroup, called containers(zone?), it's will not be bad, I think.
The final bottelneck queue for fairness in usual workload on usual (small) server will ext3's journal, I wonder ;)
Thanks,
-Kame
Quoted 124 lines
Importantly screwed! It's a very common workload pattern, and one which causes tremendous amounts of IO to be generated very quickly, traditionally causing bad latency effects all over the place. And we have no answer to this.
Quoted 111 lines
Vanilla CFQ Vs IO Controller CFQ
================================
We have not fundamentally changed CFQ, instead enhanced it to also support hierarchical io scheduling. In the process invariably there are small changes here and there as new scenarios come up. Running some tests here and comparing both the CFQ's to see if there is any major deviation in behavior.Test1: Sequential Readers
=========================
[fio --rw=read --bs=4K --size=2G --runtime=30 --direct=1 --numjobs=<1 to 16> ]IO scheduler: Vanilla CFQ
nr Max-bdwidth Min-bdwidth Agg-bdwidth Max-latency 1 35499KiB/s 35499KiB/s 35499KiB/s 19195 usec 2 17089KiB/s 13600KiB/s 30690KiB/s 118K usec 4 9165KiB/s 5421KiB/s 29411KiB/s 380K usec 8 3815KiB/s 3423KiB/s 29312KiB/s 830K usec 16 1911KiB/s 1554KiB/s 28921KiB/s 1756K usecIO scheduler: IO controller CFQ
nr Max-bdwidth Min-bdwidth Agg-bdwidth Max-latency 1 34494KiB/s 34494KiB/s 34494KiB/s 14482 usec 2 16983KiB/s 13632KiB/s 30616KiB/s 123K usec 4 9237KiB/s 5809KiB/s 29631KiB/s 372K usec 8 3901KiB/s 3505KiB/s 29162KiB/s 822K usec 16 1895KiB/s 1653KiB/s 28945KiB/s 1778K usecTest2: Sequential Writers
=========================
[fio --rw=write --bs=4K --size=2G --runtime=30 --direct=1 --numjobs=<1 to 16> ]IO scheduler: Vanilla CFQ
nr Max-bdwidth Min-bdwidth Agg-bdwidth Max-latency 1 22669KiB/s 22669KiB/s 22669KiB/s 401K usec 2 14760KiB/s 7419KiB/s 22179KiB/s 571K usec 4 5862KiB/s 5746KiB/s 23174KiB/s 444K usec 8 3377KiB/s 2199KiB/s 22427KiB/s 1057K usec 16 2229KiB/s 556KiB/s 20601KiB/s 5099K usecIO scheduler: IO Controller CFQ
nr Max-bdwidth Min-bdwidth Agg-bdwidth Max-latency 1 22911KiB/s 22911KiB/s 22911KiB/s 37319 usec 2 11752KiB/s 11632KiB/s 23383KiB/s 245K usec 4 6663KiB/s 5409KiB/s 23207KiB/s 384K usec 8 3161KiB/s 2460KiB/s 22566KiB/s 935K usec 16 1888KiB/s 795KiB/s 21349KiB/s 3009K usecTest3: Random Readers
=========================
[fio --rw=randread --bs=4K --size=2G --runtime=30 --direct=1 --numjobs=1 to 16]IO scheduler: Vanilla CFQ
nr Max-bdwidth Min-bdwidth Agg-bdwidth Max-latency 1 484KiB/s 484KiB/s 484KiB/s 22596 usec 2 229KiB/s 196KiB/s 425KiB/s 51111 usec 4 119KiB/s 73KiB/s 405KiB/s 2344 msec 8 93KiB/s 23KiB/s 399KiB/s 2246 msec 16 38KiB/s 8KiB/s 328KiB/s 3965 msecIO scheduler: IO Controller CFQ
nr Max-bdwidth Min-bdwidth Agg-bdwidth Max-latency 1 483KiB/s 483KiB/s 483KiB/s 29391 usec 2 229KiB/s 196KiB/s 426KiB/s 51625 usec 4 132KiB/s 88KiB/s 417KiB/s 2313 msec 8 79KiB/s 18KiB/s 389KiB/s 2298 msec 16 43KiB/s 9KiB/s 327KiB/s 3905 msecTest4: Random Writers
=====================
[fio --rw=randwrite --bs=4K --size=2G --runtime=30 --direct=1 --numjobs=1 to 16]IO scheduler: Vanilla CFQ
nr Max-bdwidth Min-bdwidth Agg-bdwidth Max-latency 1 14641KiB/s 14641KiB/s 14641KiB/s 93045 usec 2 7896KiB/s 1348KiB/s 9245KiB/s 82778 usec 4 2657KiB/s 265KiB/s 6025KiB/s 216K usec 8 951KiB/s 122KiB/s 3386KiB/s 1148K usec 16 66KiB/s 22KiB/s 829KiB/s 1308 msecIO scheduler: IO Controller CFQ
nr Max-bdwidth Min-bdwidth Agg-bdwidth Max-latency 1 14454KiB/s 14454KiB/s 14454KiB/s 74623 usec 2 4595KiB/s 4104KiB/s 8699KiB/s 135K usec 4 3113KiB/s 334KiB/s 5782KiB/s 200K usec 8 1146KiB/s 95KiB/s 3832KiB/s 593K usec 16 71KiB/s 29KiB/s 814KiB/s 1457 msecNotes:
- Does not look like that anything has changed significantly.Previous versions of the patches were posted here.
------------------------------------------------(V1) http://lkml.org/lkml/2009/3/11/486
(V2) http://lkml.org/lkml/2009/5/5/275
(V3) http://lkml.org/lkml/2009/5/26/472
(V4) http://lkml.org/lkml/2009/6/8/580
(V5) http://lkml.org/lkml/2009/6/19/279
(V6) http://lkml.org/lkml/2009/7/2/369
(V7) http://lkml.org/lkml/2009/7/24/253
(V8) http://lkml.org/lkml/2009/8/16/204
(V9) http://lkml.org/lkml/2009/8/28/327Quoted 2 lines
Thanks
Vivek--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Quoted 48 lines · On Fri, 25 Sep 2009 10:09:52 +0900 KAMEZAWA Hiroyuki…
Quoted 46 lines · On Thu, 24 Sep 2009 14:33:15 -0700 Andrew Morton <[email protected]>…
Quoted 22 lines
Test5 (Fairness for async writes, Buffered Write Vs Buffered Write)
===================================================================
Fairness for async writes is tricky and biggest reason is that async writes are cached in higher layers (page cahe) as well as possibly in file system layer also (btrfs, xfs etc), and are dispatched to lower layers not necessarily in proportional manner.For example, consider two dd threads reading
/dev/zeroas input file and doing writes of huge files. Very soon we will crossvm_dirty_ratioand dd thread will be forced to write out some pages to disk before more pages can be dirtied. But not necessarily dirty pages of same thread are picked. It can very well pick the inode of lesser priority dd thread and do some writeout. So effectively higher weight dd is doing writeouts of lower weight dd pages and we don't see service differentation.IOW, the core problem with buffered write fairness is that higher weight thread does not throw enought IO traffic at IO controller to keep the queue continuously backlogged. In my testing, there are many .2 to .8 second intervals where higher weight queue is empty and in that duration lower weight queue get lots of job done giving the impression that there was no service differentiation.
Quoted 5 lines
In summary, from IO controller point of view async writes support is there. Because page cache has not been designed in such a manner that higher prio/weight writer can do more write out as compared to lower prio/weight writer, gettting service differentiation is hard and it is visible in some cases and not visible in some cases.
Here's where it all falls to pieces.
For async writeback we just don't care about IO priorities. Because from the point of view of the userspace task, the write was async! It occurred at memory bandwidth speed.
It's only when the kernel's dirty memory thresholds start to get exceeded that we start to care about prioritisation. And at that time, all dirty memory (within a memcg?) is equal - a high-ioprio dirty page consumes just as much memory as a low-ioprio dirty page.
So when
balance_dirty_pages()hits, what do we want to do?I suppose that all we can do is to block low-ioprio processes more agressively at the VFS layer, to reduce the rate at which they're dirtying memory so as to give high-ioprio processes more of the disk bandwidth.
Quoted 5 lines
Quoted 3 lines
But you've gone and implemented all of this stuff at the io-controller level and not at the VFS level so you're, umm, screwed.
I think I must support dirty-ratio in memcg layer. But not yet.
OR...I'll add a bufferred-write-cgroup to track bufferred writebacks. And add a control knob as
bufferred_write.nr_dirty_thresh
to limit the number of dirty pages generetad via a cgroup.
Because memcg just records a owner of pages but not records who makes them dirty, this may be better. Maybe I can reuse page_cgroup and Ryo's blockio cgroup code.
But I'm not sure how I should treat I/Os generated out by kswapd.
Thanks,
-Kame
Quoted 51 lines · KAMEZAWA Hiroyuki
Quoted 48 lines · On Fri, 25 Sep 2009 10:09:52 +0900 KAMEZAWA Hiroyuki…
Quoted 46 lines · On Thu, 24 Sep 2009 14:33:15 -0700 Andrew Morton <[email protected]>…
Quoted 22 lines
Test5 (Fairness for async writes, Buffered Write Vs Buffered Write)
===================================================================
Fairness for async writes is tricky and biggest reason is that async writes are cached in higher layers (page cahe) as well as possibly in file system layer also (btrfs, xfs etc), and are dispatched to lower layers not necessarily in proportional manner.For example, consider two dd threads reading
/dev/zeroas input file and doing writes of huge files. Very soon we will crossvm_dirty_ratioand dd thread will be forced to write out some pages to disk before more pages can be dirtied. But not necessarily dirty pages of same thread are picked. It can very well pick the inode of lesser priority dd thread and do some writeout. So effectively higher weight dd is doing writeouts of lower weight dd pages and we don't see service differentation.IOW, the core problem with buffered write fairness is that higher weight thread does not throw enought IO traffic at IO controller to keep the queue continuously backlogged. In my testing, there are many .2 to .8 second intervals where higher weight queue is empty and in that duration lower weight queue get lots of job done giving the impression that there was no service differentiation.
Quoted 5 lines
In summary, from IO controller point of view async writes support is there. Because page cache has not been designed in such a manner that higher prio/weight writer can do more write out as compared to lower prio/weight writer, gettting service differentiation is hard and it is visible in some cases and not visible in some cases.
Here's where it all falls to pieces.
For async writeback we just don't care about IO priorities. Because from the point of view of the userspace task, the write was async! It occurred at memory bandwidth speed.
It's only when the kernel's dirty memory thresholds start to get exceeded that we start to care about prioritisation. And at that time, all dirty memory (within a memcg?) is equal - a high-ioprio dirty page consumes just as much memory as a low-ioprio dirty page.
So when
balance_dirty_pages()hits, what do we want to do?I suppose that all we can do is to block low-ioprio processes more agressively at the VFS layer, to reduce the rate at which they're dirtying memory so as to give high-ioprio processes more of the disk bandwidth.
Quoted 6 lines
Quoted 5 lines
Quoted 3 lines
But you've gone and implemented all of this stuff at the io-controller level and not at the VFS level so you're, umm, screwed.
I think I must support dirty-ratio in memcg layer. But not yet.
We need to add this to the TODO list.
Quoted 8 lines
OR...I'll add a bufferred-write-cgroup to track bufferred writebacks. And add a control knob as
bufferred_write.nr_dirty_thresh
to limit the number of dirty pages generetad via a cgroup.Because memcg just records a owner of pages but not records who makes them dirty, this may be better. Maybe I can reuse
page_cgroupand Ryo's blockio cgroup code.
Very good point, this is crucial for shared pages.
Quoted 3 lines
But I'm not sure how I should treat I/Os generated out by kswapd.
Account them to process 0 :)
Signature · 1 line
Balbir
Hi,
Quoted 14 lines · Balbir Singh
Quoted 2 lines
Quoted 1 line
I think I must support dirty-ratio in memcg layer. But not yet.
We need to add this to the TODO list.
Quoted 8 lines
OR...I'll add a bufferred-write-cgroup to track bufferred writebacks. And add a control knob as
bufferred_write.nr_dirty_thresh
to limit the number of dirty pages generetad via a cgroup.Because memcg just records a owner of pages but not records who makes them dirty, this may be better. Maybe I can reuse
page_cgroupand Ryo's blockio cgroup code.
Quoted 7 lines
Very good point, this is crucial for shared pages.
Quoted 3 lines
But I'm not sure how I should treat I/Os generated out by kswapd.
Account them to process 0 :)
How about accounting them to processes who make pages dirty? I think that a process which consumes more memory should get penalty. However, this allows a page request process to use other's bandwidth, but If a user doesn't want to swap-out the memory, the user should allocate enough memory for the process by using memcg in advance.
Thanks,
Ryo Tsuruta
A message that is not in the archive (sent off-list or not yet received): 20090925052911.GK4590-SINUvgVNF2CyUtPGxGje5AC/[email protected]
Hi,
Quoted 14 lines · Balbir Singh
Quoted 2 lines
Quoted 1 line
I think I must support dirty-ratio in memcg layer. But not yet.
We need to add this to the TODO list.
Quoted 8 lines
OR...I'll add a bufferred-write-cgroup to track bufferred writebacks. And add a control knob as
bufferred_write.nr_dirty_thresh
to limit the number of dirty pages generetad via a cgroup.Because memcg just records a owner of pages but not records who makes them dirty, this may be better. Maybe I can reuse
page_cgroupand Ryo's blockio cgroup code.
Quoted 7 lines
Very good point, this is crucial for shared pages.
Quoted 3 lines
But I'm not sure how I should treat I/Os generated out by kswapd.
Account them to process 0 :)
How about accounting them to processes who make pages dirty? I think that a process which consumes more memory should get penalty. However, this allows a page request process to use other's bandwidth, but If a user doesn't want to swap-out the memory, the user should allocate enough memory for the process by using memcg in advance.
Thanks,
Ryo Tsuruta
A message that is not in the archive (sent off-list or not yet received): 20090925101821.1de8091a.kamezawa.hiroyu-+CUm20s59erQFUHtdCDX3A@public.gmane.org
Quoted 51 lines · KAMEZAWA Hiroyuki
Quoted 48 lines · On Fri, 25 Sep 2009 10:09:52 +0900 KAMEZAWA Hiroyuki…
Quoted 46 lines · On Thu, 24 Sep 2009 14:33:15 -0700 Andrew Morton…
Quoted 22 lines
Test5 (Fairness for async writes, Buffered Write Vs Buffered Write)
===================================================================
Fairness for async writes is tricky and biggest reason is that async writes are cached in higher layers (page cahe) as well as possibly in file system layer also (btrfs, xfs etc), and are dispatched to lower layers not necessarily in proportional manner.For example, consider two dd threads reading
/dev/zeroas input file and doing writes of huge files. Very soon we will crossvm_dirty_ratioand dd thread will be forced to write out some pages to disk before more pages can be dirtied. But not necessarily dirty pages of same thread are picked. It can very well pick the inode of lesser priority dd thread and do some writeout. So effectively higher weight dd is doing writeouts of lower weight dd pages and we don't see service differentation.IOW, the core problem with buffered write fairness is that higher weight thread does not throw enought IO traffic at IO controller to keep the queue continuously backlogged. In my testing, there are many .2 to .8 second intervals where higher weight queue is empty and in that duration lower weight queue get lots of job done giving the impression that there was no service differentiation.
Quoted 5 lines
In summary, from IO controller point of view async writes support is there. Because page cache has not been designed in such a manner that higher prio/weight writer can do more write out as compared to lower prio/weight writer, gettting service differentiation is hard and it is visible in some cases and not visible in some cases.
Here's where it all falls to pieces.
For async writeback we just don't care about IO priorities. Because from the point of view of the userspace task, the write was async! It occurred at memory bandwidth speed.
It's only when the kernel's dirty memory thresholds start to get exceeded that we start to care about prioritisation. And at that time, all dirty memory (within a memcg?) is equal - a high-ioprio dirty page consumes just as much memory as a low-ioprio dirty page.
So when
balance_dirty_pages()hits, what do we want to do?I suppose that all we can do is to block low-ioprio processes more agressively at the VFS layer, to reduce the rate at which they're dirtying memory so as to give high-ioprio processes more of the disk bandwidth.
Quoted 6 lines
Quoted 5 lines
Quoted 3 lines
But you've gone and implemented all of this stuff at the io-controller level and not at the VFS level so you're, umm, screwed.
I think I must support dirty-ratio in memcg layer. But not yet.
We need to add this to the TODO list.
Quoted 8 lines
OR...I'll add a bufferred-write-cgroup to track bufferred writebacks. And add a control knob as
bufferred_write.nr_dirty_thresh
to limit the number of dirty pages generetad via a cgroup.Because memcg just records a owner of pages but not records who makes them dirty, this may be better. Maybe I can reuse
page_cgroupand Ryo's blockio cgroup code.
Very good point, this is crucial for shared pages.
Quoted 3 lines
But I'm not sure how I should treat I/Os generated out by kswapd.
Account them to process 0 :)
Signature · 1 line
Balbir
A message that is not in the archive (sent off-list or not yet received): 20090925100952.55c2dd7a.kamezawa.hiroyu-+CUm20s59erQFUHtdCDX3A@public.gmane.org
Quoted 48 lines · On Fri, 25 Sep 2009 10:09:52 +0900 KAMEZAWA Hiroyuki…
Quoted 46 lines · On Thu, 24 Sep 2009 14:33:15 -0700 Andrew Morton…
Quoted 22 lines
Test5 (Fairness for async writes, Buffered Write Vs Buffered Write)
===================================================================
Fairness for async writes is tricky and biggest reason is that async writes are cached in higher layers (page cahe) as well as possibly in file system layer also (btrfs, xfs etc), and are dispatched to lower layers not necessarily in proportional manner.For example, consider two dd threads reading
/dev/zeroas input file and doing writes of huge files. Very soon we will crossvm_dirty_ratioand dd thread will be forced to write out some pages to disk before more pages can be dirtied. But not necessarily dirty pages of same thread are picked. It can very well pick the inode of lesser priority dd thread and do some writeout. So effectively higher weight dd is doing writeouts of lower weight dd pages and we don't see service differentation.IOW, the core problem with buffered write fairness is that higher weight thread does not throw enought IO traffic at IO controller to keep the queue continuously backlogged. In my testing, there are many .2 to .8 second intervals where higher weight queue is empty and in that duration lower weight queue get lots of job done giving the impression that there was no service differentiation.
Quoted 5 lines
In summary, from IO controller point of view async writes support is there. Because page cache has not been designed in such a manner that higher prio/weight writer can do more write out as compared to lower prio/weight writer, gettting service differentiation is hard and it is visible in some cases and not visible in some cases.
Here's where it all falls to pieces.
For async writeback we just don't care about IO priorities. Because from the point of view of the userspace task, the write was async! It occurred at memory bandwidth speed.
It's only when the kernel's dirty memory thresholds start to get exceeded that we start to care about prioritisation. And at that time, all dirty memory (within a memcg?) is equal - a high-ioprio dirty page consumes just as much memory as a low-ioprio dirty page.
So when
balance_dirty_pages()hits, what do we want to do?I suppose that all we can do is to block low-ioprio processes more agressively at the VFS layer, to reduce the rate at which they're dirtying memory so as to give high-ioprio processes more of the disk bandwidth.
Quoted 5 lines
Quoted 3 lines
But you've gone and implemented all of this stuff at the io-controller level and not at the VFS level so you're, umm, screwed.
I think I must support dirty-ratio in memcg layer. But not yet.
OR...I'll add a bufferred-write-cgroup to track bufferred writebacks. And add a control knob as
bufferred_write.nr_dirty_thresh
to limit the number of dirty pages generetad via a cgroup.
Because memcg just records a owner of pages but not records who makes them dirty, this may be better. Maybe I can reuse page_cgroup and Ryo's blockio cgroup code.
But I'm not sure how I should treat I/Os generated out by kswapd.
Thanks,
-Kame
Quoted 48 lines · KAMEZAWA Hiroyuki
Quoted 46 lines · On Thu, 24 Sep 2009 14:33:15 -0700 Andrew Morton…
Quoted 22 lines
Test5 (Fairness for async writes, Buffered Write Vs Buffered Write)
===================================================================
Fairness for async writes is tricky and biggest reason is that async writes are cached in higher layers (page cahe) as well as possibly in file system layer also (btrfs, xfs etc), and are dispatched to lower layers not necessarily in proportional manner.For example, consider two dd threads reading
/dev/zeroas input file and doing writes of huge files. Very soon we will crossvm_dirty_ratioand dd thread will be forced to write out some pages to disk before more pages can be dirtied. But not necessarily dirty pages of same thread are picked. It can very well pick the inode of lesser priority dd thread and do some writeout. So effectively higher weight dd is doing writeouts of lower weight dd pages and we don't see service differentation.IOW, the core problem with buffered write fairness is that higher weight thread does not throw enought IO traffic at IO controller to keep the queue continuously backlogged. In my testing, there are many .2 to .8 second intervals where higher weight queue is empty and in that duration lower weight queue get lots of job done giving the impression that there was no service differentiation.
Quoted 5 lines
In summary, from IO controller point of view async writes support is there. Because page cache has not been designed in such a manner that higher prio/weight writer can do more write out as compared to lower prio/weight writer, gettting service differentiation is hard and it is visible in some cases and not visible in some cases.
Here's where it all falls to pieces.
For async writeback we just don't care about IO priorities. Because from the point of view of the userspace task, the write was async! It occurred at memory bandwidth speed.
It's only when the kernel's dirty memory thresholds start to get exceeded that we start to care about prioritisation. And at that time, all dirty memory (within a memcg?) is equal - a high-ioprio dirty page consumes just as much memory as a low-ioprio dirty page.
So when
balance_dirty_pages()hits, what do we want to do?I suppose that all we can do is to block low-ioprio processes more agressively at the VFS layer, to reduce the rate at which they're dirtying memory so as to give high-ioprio processes more of the disk bandwidth.
Quoted 7 lines
Quoted 3 lines
But you've gone and implemented all of this stuff at the io-controller level and not at the VFS level so you're, umm, screwed.
I think I must support dirty-ratio in memcg layer. But not yet. I can't easily imagine how the system will work if both dirty-ratio and io-controller cgroup are supported.
IIUC, you are suggesting per memeory cgroup dirty ratio and writer will be throttled if dirty ratio is crossed. makes sense to me. Just that io controller and memory controller shall have to me mounted together.
Thanks
Vivek
Quoted 134 lines
But considering use them as a set of
cgroup, called containers(zone?), it's will not be bad, I think.The final bottelneck queue for fairness in usual workload on usual (small) server will ext3's journal, I wonder ;)
Thanks,
-KameQuoted 124 lines
Importantly screwed! It's a very common workload pattern, and one which causes tremendous amounts of IO to be generated very quickly, traditionally causing bad latency effects all over the place. And we have no answer to this.
Quoted 111 lines
Vanilla CFQ Vs IO Controller CFQ
================================
We have not fundamentally changed CFQ, instead enhanced it to also support hierarchical io scheduling. In the process invariably there are small changes here and there as new scenarios come up. Running some tests here and comparing both the CFQ's to see if there is any major deviation in behavior.Test1: Sequential Readers
=========================
[fio --rw=read --bs=4K --size=2G --runtime=30 --direct=1 --numjobs=<1 to 16> ]IO scheduler: Vanilla CFQ
nr Max-bdwidth Min-bdwidth Agg-bdwidth Max-latency 1 35499KiB/s 35499KiB/s 35499KiB/s 19195 usec 2 17089KiB/s 13600KiB/s 30690KiB/s 118K usec 4 9165KiB/s 5421KiB/s 29411KiB/s 380K usec 8 3815KiB/s 3423KiB/s 29312KiB/s 830K usec 16 1911KiB/s 1554KiB/s 28921KiB/s 1756K usecIO scheduler: IO controller CFQ
nr Max-bdwidth Min-bdwidth Agg-bdwidth Max-latency 1 34494KiB/s 34494KiB/s 34494KiB/s 14482 usec 2 16983KiB/s 13632KiB/s 30616KiB/s 123K usec 4 9237KiB/s 5809KiB/s 29631KiB/s 372K usec 8 3901KiB/s 3505KiB/s 29162KiB/s 822K usec 16 1895KiB/s 1653KiB/s 28945KiB/s 1778K usecTest2: Sequential Writers
=========================
[fio --rw=write --bs=4K --size=2G --runtime=30 --direct=1 --numjobs=<1 to 16> ]IO scheduler: Vanilla CFQ
nr Max-bdwidth Min-bdwidth Agg-bdwidth Max-latency 1 22669KiB/s 22669KiB/s 22669KiB/s 401K usec 2 14760KiB/s 7419KiB/s 22179KiB/s 571K usec 4 5862KiB/s 5746KiB/s 23174KiB/s 444K usec 8 3377KiB/s 2199KiB/s 22427KiB/s 1057K usec 16 2229KiB/s 556KiB/s 20601KiB/s 5099K usecIO scheduler: IO Controller CFQ
nr Max-bdwidth Min-bdwidth Agg-bdwidth Max-latency 1 22911KiB/s 22911KiB/s 22911KiB/s 37319 usec 2 11752KiB/s 11632KiB/s 23383KiB/s 245K usec 4 6663KiB/s 5409KiB/s 23207KiB/s 384K usec 8 3161KiB/s 2460KiB/s 22566KiB/s 935K usec 16 1888KiB/s 795KiB/s 21349KiB/s 3009K usecTest3: Random Readers
=========================
[fio --rw=randread --bs=4K --size=2G --runtime=30 --direct=1 --numjobs=1 to 16]IO scheduler: Vanilla CFQ
nr Max-bdwidth Min-bdwidth Agg-bdwidth Max-latency 1 484KiB/s 484KiB/s 484KiB/s 22596 usec 2 229KiB/s 196KiB/s 425KiB/s 51111 usec 4 119KiB/s 73KiB/s 405KiB/s 2344 msec 8 93KiB/s 23KiB/s 399KiB/s 2246 msec 16 38KiB/s 8KiB/s 328KiB/s 3965 msecIO scheduler: IO Controller CFQ
nr Max-bdwidth Min-bdwidth Agg-bdwidth Max-latency 1 483KiB/s 483KiB/s 483KiB/s 29391 usec 2 229KiB/s 196KiB/s 426KiB/s 51625 usec 4 132KiB/s 88KiB/s 417KiB/s 2313 msec 8 79KiB/s 18KiB/s 389KiB/s 2298 msec 16 43KiB/s 9KiB/s 327KiB/s 3905 msecTest4: Random Writers
=====================
[fio --rw=randwrite --bs=4K --size=2G --runtime=30 --direct=1 --numjobs=1 to 16]IO scheduler: Vanilla CFQ
nr Max-bdwidth Min-bdwidth Agg-bdwidth Max-latency 1 14641KiB/s 14641KiB/s 14641KiB/s 93045 usec 2 7896KiB/s 1348KiB/s 9245KiB/s 82778 usec 4 2657KiB/s 265KiB/s 6025KiB/s 216K usec 8 951KiB/s 122KiB/s 3386KiB/s 1148K usec 16 66KiB/s 22KiB/s 829KiB/s 1308 msecIO scheduler: IO Controller CFQ
nr Max-bdwidth Min-bdwidth Agg-bdwidth Max-latency 1 14454KiB/s 14454KiB/s 14454KiB/s 74623 usec 2 4595KiB/s 4104KiB/s 8699KiB/s 135K usec 4 3113KiB/s 334KiB/s 5782KiB/s 200K usec 8 1146KiB/s 95KiB/s 3832KiB/s 593K usec 16 71KiB/s 29KiB/s 814KiB/s 1457 msecNotes:
- Does not look like that anything has changed significantly.Previous versions of the patches were posted here.
------------------------------------------------(V1) http://lkml.org/lkml/2009/3/11/486
(V2) http://lkml.org/lkml/2009/5/5/275
(V3) http://lkml.org/lkml/2009/5/26/472
(V4) http://lkml.org/lkml/2009/6/8/580
(V5) http://lkml.org/lkml/2009/6/19/279
(V6) http://lkml.org/lkml/2009/7/2/369
(V7) http://lkml.org/lkml/2009/7/24/253
(V8) http://lkml.org/lkml/2009/8/16/204
(V9) http://lkml.org/lkml/2009/8/28/327Quoted 2 lines
Thanks
Vivek--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Quoted 48 lines · KAMEZAWA Hiroyuki
Quoted 46 lines · On Thu, 24 Sep 2009 14:33:15 -0700 Andrew Morton <[email protected]>…
Quoted 22 lines
Test5 (Fairness for async writes, Buffered Write Vs Buffered Write)
===================================================================
Fairness for async writes is tricky and biggest reason is that async writes are cached in higher layers (page cahe) as well as possibly in file system layer also (btrfs, xfs etc), and are dispatched to lower layers not necessarily in proportional manner.For example, consider two dd threads reading
/dev/zeroas input file and doing writes of huge files. Very soon we will crossvm_dirty_ratioand dd thread will be forced to write out some pages to disk before more pages can be dirtied. But not necessarily dirty pages of same thread are picked. It can very well pick the inode of lesser priority dd thread and do some writeout. So effectively higher weight dd is doing writeouts of lower weight dd pages and we don't see service differentation.IOW, the core problem with buffered write fairness is that higher weight thread does not throw enought IO traffic at IO controller to keep the queue continuously backlogged. In my testing, there are many .2 to .8 second intervals where higher weight queue is empty and in that duration lower weight queue get lots of job done giving the impression that there was no service differentiation.
Quoted 5 lines
In summary, from IO controller point of view async writes support is there. Because page cache has not been designed in such a manner that higher prio/weight writer can do more write out as compared to lower prio/weight writer, gettting service differentiation is hard and it is visible in some cases and not visible in some cases.
Here's where it all falls to pieces.
For async writeback we just don't care about IO priorities. Because from the point of view of the userspace task, the write was async! It occurred at memory bandwidth speed.
It's only when the kernel's dirty memory thresholds start to get exceeded that we start to care about prioritisation. And at that time, all dirty memory (within a memcg?) is equal - a high-ioprio dirty page consumes just as much memory as a low-ioprio dirty page.
So when
balance_dirty_pages()hits, what do we want to do?I suppose that all we can do is to block low-ioprio processes more agressively at the VFS layer, to reduce the rate at which they're dirtying memory so as to give high-ioprio processes more of the disk bandwidth.
Quoted 7 lines
Quoted 3 lines
But you've gone and implemented all of this stuff at the io-controller level and not at the VFS level so you're, umm, screwed.
I think I must support dirty-ratio in memcg layer. But not yet. I can't easily imagine how the system will work if both dirty-ratio and io-controller cgroup are supported.
IIUC, you are suggesting per memeory cgroup dirty ratio and writer will be throttled if dirty ratio is crossed. makes sense to me. Just that io controller and memory controller shall have to me mounted together.
Thanks
Vivek
Quoted 134 lines
But considering use them as a set of
cgroup, called containers(zone?), it's will not be bad, I think.The final bottelneck queue for fairness in usual workload on usual (small) server will ext3's journal, I wonder ;)
Thanks,
-KameQuoted 124 lines
Importantly screwed! It's a very common workload pattern, and one which causes tremendous amounts of IO to be generated very quickly, traditionally causing bad latency effects all over the place. And we have no answer to this.
Quoted 111 lines
Vanilla CFQ Vs IO Controller CFQ
================================
We have not fundamentally changed CFQ, instead enhanced it to also support hierarchical io scheduling. In the process invariably there are small changes here and there as new scenarios come up. Running some tests here and comparing both the CFQ's to see if there is any major deviation in behavior.Test1: Sequential Readers
=========================
[fio --rw=read --bs=4K --size=2G --runtime=30 --direct=1 --numjobs=<1 to 16> ]IO scheduler: Vanilla CFQ
nr Max-bdwidth Min-bdwidth Agg-bdwidth Max-latency 1 35499KiB/s 35499KiB/s 35499KiB/s 19195 usec 2 17089KiB/s 13600KiB/s 30690KiB/s 118K usec 4 9165KiB/s 5421KiB/s 29411KiB/s 380K usec 8 3815KiB/s 3423KiB/s 29312KiB/s 830K usec 16 1911KiB/s 1554KiB/s 28921KiB/s 1756K usecIO scheduler: IO controller CFQ
nr Max-bdwidth Min-bdwidth Agg-bdwidth Max-latency 1 34494KiB/s 34494KiB/s 34494KiB/s 14482 usec 2 16983KiB/s 13632KiB/s 30616KiB/s 123K usec 4 9237KiB/s 5809KiB/s 29631KiB/s 372K usec 8 3901KiB/s 3505KiB/s 29162KiB/s 822K usec 16 1895KiB/s 1653KiB/s 28945KiB/s 1778K usecTest2: Sequential Writers
=========================
[fio --rw=write --bs=4K --size=2G --runtime=30 --direct=1 --numjobs=<1 to 16> ]IO scheduler: Vanilla CFQ
nr Max-bdwidth Min-bdwidth Agg-bdwidth Max-latency 1 22669KiB/s 22669KiB/s 22669KiB/s 401K usec 2 14760KiB/s 7419KiB/s 22179KiB/s 571K usec 4 5862KiB/s 5746KiB/s 23174KiB/s 444K usec 8 3377KiB/s 2199KiB/s 22427KiB/s 1057K usec 16 2229KiB/s 556KiB/s 20601KiB/s 5099K usecIO scheduler: IO Controller CFQ
nr Max-bdwidth Min-bdwidth Agg-bdwidth Max-latency 1 22911KiB/s 22911KiB/s 22911KiB/s 37319 usec 2 11752KiB/s 11632KiB/s 23383KiB/s 245K usec 4 6663KiB/s 5409KiB/s 23207KiB/s 384K usec 8 3161KiB/s 2460KiB/s 22566KiB/s 935K usec 16 1888KiB/s 795KiB/s 21349KiB/s 3009K usecTest3: Random Readers
=========================
[fio --rw=randread --bs=4K --size=2G --runtime=30 --direct=1 --numjobs=1 to 16]IO scheduler: Vanilla CFQ
nr Max-bdwidth Min-bdwidth Agg-bdwidth Max-latency 1 484KiB/s 484KiB/s 484KiB/s 22596 usec 2 229KiB/s 196KiB/s 425KiB/s 51111 usec 4 119KiB/s 73KiB/s 405KiB/s 2344 msec 8 93KiB/s 23KiB/s 399KiB/s 2246 msec 16 38KiB/s 8KiB/s 328KiB/s 3965 msecIO scheduler: IO Controller CFQ
nr Max-bdwidth Min-bdwidth Agg-bdwidth Max-latency 1 483KiB/s 483KiB/s 483KiB/s 29391 usec 2 229KiB/s 196KiB/s 426KiB/s 51625 usec 4 132KiB/s 88KiB/s 417KiB/s 2313 msec 8 79KiB/s 18KiB/s 389KiB/s 2298 msec 16 43KiB/s 9KiB/s 327KiB/s 3905 msecTest4: Random Writers
=====================
[fio --rw=randwrite --bs=4K --size=2G --runtime=30 --direct=1 --numjobs=1 to 16]IO scheduler: Vanilla CFQ
nr Max-bdwidth Min-bdwidth Agg-bdwidth Max-latency 1 14641KiB/s 14641KiB/s 14641KiB/s 93045 usec 2 7896KiB/s 1348KiB/s 9245KiB/s 82778 usec 4 2657KiB/s 265KiB/s 6025KiB/s 216K usec 8 951KiB/s 122KiB/s 3386KiB/s 1148K usec 16 66KiB/s 22KiB/s 829KiB/s 1308 msecIO scheduler: IO Controller CFQ
nr Max-bdwidth Min-bdwidth Agg-bdwidth Max-latency 1 14454KiB/s 14454KiB/s 14454KiB/s 74623 usec 2 4595KiB/s 4104KiB/s 8699KiB/s 135K usec 4 3113KiB/s 334KiB/s 5782KiB/s 200K usec 8 1146KiB/s 95KiB/s 3832KiB/s 593K usec 16 71KiB/s 29KiB/s 814KiB/s 1457 msecNotes:
- Does not look like that anything has changed significantly.Previous versions of the patches were posted here.
------------------------------------------------(V1) http://lkml.org/lkml/2009/3/11/486
(V2) http://lkml.org/lkml/2009/5/5/275
(V3) http://lkml.org/lkml/2009/5/26/472
(V4) http://lkml.org/lkml/2009/6/8/580
(V5) http://lkml.org/lkml/2009/6/19/279
(V6) http://lkml.org/lkml/2009/7/2/369
(V7) http://lkml.org/lkml/2009/7/24/253
(V8) http://lkml.org/lkml/2009/8/16/204
(V9) http://lkml.org/lkml/2009/8/28/327Quoted 2 lines
Thanks
Vivek--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
A message that is not in the archive (sent off-list or not yet received): 20090924143315.781cd0ac.akpm-de/[email protected]
Quoted 41 lines · On Thu, 24 Sep 2009 14:33:15 -0700 Andrew Morton…
Quoted 22 lines
Test5 (Fairness for async writes, Buffered Write Vs Buffered Write)
===================================================================
Fairness for async writes is tricky and biggest reason is that async writes are cached in higher layers (page cahe) as well as possibly in file system layer also (btrfs, xfs etc), and are dispatched to lower layers not necessarily in proportional manner.For example, consider two dd threads reading
/dev/zeroas input file and doing writes of huge files. Very soon we will crossvm_dirty_ratioand dd thread will be forced to write out some pages to disk before more pages can be dirtied. But not necessarily dirty pages of same thread are picked. It can very well pick the inode of lesser priority dd thread and do some writeout. So effectively higher weight dd is doing writeouts of lower weight dd pages and we don't see service differentation.IOW, the core problem with buffered write fairness is that higher weight thread does not throw enought IO traffic at IO controller to keep the queue continuously backlogged. In my testing, there are many .2 to .8 second intervals where higher weight queue is empty and in that duration lower weight queue get lots of job done giving the impression that there was no service differentiation.
Quoted 5 lines
In summary, from IO controller point of view async writes support is there. Because page cache has not been designed in such a manner that higher prio/weight writer can do more write out as compared to lower prio/weight writer, gettting service differentiation is hard and it is visible in some cases and not visible in some cases.
Here's where it all falls to pieces.
For async writeback we just don't care about IO priorities. Because from the point of view of the userspace task, the write was async! It occurred at memory bandwidth speed.
It's only when the kernel's dirty memory thresholds start to get exceeded that we start to care about prioritisation. And at that time, all dirty memory (within a memcg?) is equal - a high-ioprio dirty page consumes just as much memory as a low-ioprio dirty page.
So when
balance_dirty_pages()hits, what do we want to do?
Quoted 8 lines
I suppose that all we can do is to block low-ioprio processes more agressively at the VFS layer, to reduce the rate at which they're dirtying memory so as to give high-ioprio processes more of the disk bandwidth.
But you've gone and implemented all of this stuff at the io-controller level and not at the VFS level so you're, umm, screwed.
I think I must support dirty-ratio in memcg layer. But not yet. I can't easily imagine how the system will work if both dirty-ratio and io-controller cgroup are supported. But considering use them as a set of cgroup, called containers(zone?), it's will not be bad, I think.
The final bottelneck queue for fairness in usual workload on usual (small) server will ext3's journal, I wonder ;)
Thanks,
-Kame
Quoted 124 lines
Importantly screwed! It's a very common workload pattern, and one which causes tremendous amounts of IO to be generated very quickly, traditionally causing bad latency effects all over the place. And we have no answer to this.
Quoted 111 lines
Vanilla CFQ Vs IO Controller CFQ
================================
We have not fundamentally changed CFQ, instead enhanced it to also support hierarchical io scheduling. In the process invariably there are small changes here and there as new scenarios come up. Running some tests here and comparing both the CFQ's to see if there is any major deviation in behavior.Test1: Sequential Readers
=========================
[fio --rw=read --bs=4K --size=2G --runtime=30 --direct=1 --numjobs=<1 to 16> ]IO scheduler: Vanilla CFQ
nr Max-bdwidth Min-bdwidth Agg-bdwidth Max-latency 1 35499KiB/s 35499KiB/s 35499KiB/s 19195 usec 2 17089KiB/s 13600KiB/s 30690KiB/s 118K usec 4 9165KiB/s 5421KiB/s 29411KiB/s 380K usec 8 3815KiB/s 3423KiB/s 29312KiB/s 830K usec 16 1911KiB/s 1554KiB/s 28921KiB/s 1756K usecIO scheduler: IO controller CFQ
nr Max-bdwidth Min-bdwidth Agg-bdwidth Max-latency 1 34494KiB/s 34494KiB/s 34494KiB/s 14482 usec 2 16983KiB/s 13632KiB/s 30616KiB/s 123K usec 4 9237KiB/s 5809KiB/s 29631KiB/s 372K usec 8 3901KiB/s 3505KiB/s 29162KiB/s 822K usec 16 1895KiB/s 1653KiB/s 28945KiB/s 1778K usecTest2: Sequential Writers
=========================
[fio --rw=write --bs=4K --size=2G --runtime=30 --direct=1 --numjobs=<1 to 16> ]IO scheduler: Vanilla CFQ
nr Max-bdwidth Min-bdwidth Agg-bdwidth Max-latency 1 22669KiB/s 22669KiB/s 22669KiB/s 401K usec 2 14760KiB/s 7419KiB/s 22179KiB/s 571K usec 4 5862KiB/s 5746KiB/s 23174KiB/s 444K usec 8 3377KiB/s 2199KiB/s 22427KiB/s 1057K usec 16 2229KiB/s 556KiB/s 20601KiB/s 5099K usecIO scheduler: IO Controller CFQ
nr Max-bdwidth Min-bdwidth Agg-bdwidth Max-latency 1 22911KiB/s 22911KiB/s 22911KiB/s 37319 usec 2 11752KiB/s 11632KiB/s 23383KiB/s 245K usec 4 6663KiB/s 5409KiB/s 23207KiB/s 384K usec 8 3161KiB/s 2460KiB/s 22566KiB/s 935K usec 16 1888KiB/s 795KiB/s 21349KiB/s 3009K usecTest3: Random Readers
=========================
[fio --rw=randread --bs=4K --size=2G --runtime=30 --direct=1 --numjobs=1 to 16]IO scheduler: Vanilla CFQ
nr Max-bdwidth Min-bdwidth Agg-bdwidth Max-latency 1 484KiB/s 484KiB/s 484KiB/s 22596 usec 2 229KiB/s 196KiB/s 425KiB/s 51111 usec 4 119KiB/s 73KiB/s 405KiB/s 2344 msec 8 93KiB/s 23KiB/s 399KiB/s 2246 msec 16 38KiB/s 8KiB/s 328KiB/s 3965 msecIO scheduler: IO Controller CFQ
nr Max-bdwidth Min-bdwidth Agg-bdwidth Max-latency 1 483KiB/s 483KiB/s 483KiB/s 29391 usec 2 229KiB/s 196KiB/s 426KiB/s 51625 usec 4 132KiB/s 88KiB/s 417KiB/s 2313 msec 8 79KiB/s 18KiB/s 389KiB/s 2298 msec 16 43KiB/s 9KiB/s 327KiB/s 3905 msecTest4: Random Writers
=====================
[fio --rw=randwrite --bs=4K --size=2G --runtime=30 --direct=1 --numjobs=1 to 16]IO scheduler: Vanilla CFQ
nr Max-bdwidth Min-bdwidth Agg-bdwidth Max-latency 1 14641KiB/s 14641KiB/s 14641KiB/s 93045 usec 2 7896KiB/s 1348KiB/s 9245KiB/s 82778 usec 4 2657KiB/s 265KiB/s 6025KiB/s 216K usec 8 951KiB/s 122KiB/s 3386KiB/s 1148K usec 16 66KiB/s 22KiB/s 829KiB/s 1308 msecIO scheduler: IO Controller CFQ
nr Max-bdwidth Min-bdwidth Agg-bdwidth Max-latency 1 14454KiB/s 14454KiB/s 14454KiB/s 74623 usec 2 4595KiB/s 4104KiB/s 8699KiB/s 135K usec 4 3113KiB/s 334KiB/s 5782KiB/s 200K usec 8 1146KiB/s 95KiB/s 3832KiB/s 593K usec 16 71KiB/s 29KiB/s 814KiB/s 1457 msecNotes:
- Does not look like that anything has changed significantly.Previous versions of the patches were posted here.
------------------------------------------------(V1) http://lkml.org/lkml/2009/3/11/486
(V2) http://lkml.org/lkml/2009/5/5/275
(V3) http://lkml.org/lkml/2009/5/26/472
(V4) http://lkml.org/lkml/2009/6/8/580
(V5) http://lkml.org/lkml/2009/6/19/279
(V6) http://lkml.org/lkml/2009/7/2/369
(V7) http://lkml.org/lkml/2009/7/24/253
(V8) http://lkml.org/lkml/2009/8/16/204
(V9) http://lkml.org/lkml/2009/8/28/327Quoted 2 lines
Thanks
Vivek--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Quoted 104 lines · Andrew Morton
Quoted 5 lines · On Thu, 24 Sep 2009 15:25:04 -0400 Vivek Goyal…
Hi All,
Here is the V10 of the IO controller patches generated on top of 2.6.31.
Thanks for the writeup. It really helps and is most worthwhile for a project of this importance, size and complexity.
Quoted 76 lines
What problem are we trying to solve
===================================
Provide group IO scheduling feature in Linux along the lines of other resource controllers like cpu.IOW, provide facility so that a user can group applications using cgroups and control the amount of disk time/bandwidth received by a group based on its weight.
How to solve the problem
=========================Different people have solved the issue differetnly. So far looks it looks like we seem to have following two core requirements when it comes to fairness at group level.
- Control bandwidth seen by groups.
- Control on latencies when a request gets backlogged in group.At least there are now three patchsets available (including this one).
IO throttling
-------------
This is a bandwidth controller which keeps track of IO rate of a group and throttles the process in the group if it exceeds the user specified limit.dm-ioband
---------
This is a proportional bandwidth controller implemented as device mapper driver and provides fair access in terms of amount of IO done (not in terms of disk time as CFQ does).So one will setup one or more dm-ioband devices on top of physical/logical block device, configure the ioband device and pass information like grouping etc. Now this device will keep track of bios flowing through it and control the flow of bios based on group policies.
IO scheduler based IO controller
--------------------------------
Here we have viewed the problem of IO contoller as hierarchical group scheduling (along the lines of CFS group scheduling) issue. Currently one can view linux IO schedulers as flat where there is one root group and all the IO belongs to that group.This patchset basically modifies IO schedulers to also support hierarchical group scheduling. CFQ already provides fairness among different processes. I have extended it support group IO schduling. Also took some of the code out of CFQ and put in a common layer so that same group scheduling code can be used by noop, deadline and AS to support group scheduling.
Pros/Cons
=========
There are pros and cons to each of the approach. Following are some of the thoughts.Max bandwidth vs proportional bandwidth
---------------------------------------
IO throttling is a max bandwidth controller and not a proportional one. Additionaly it provides fairness in terms of amount of IO done (and not in terms of disk time as CFQ does).Personally, I think that proportional weight controller is useful to more people than just max bandwidth controller. In addition, IO scheduler based controller can also be enhanced to do max bandwidth control. So it can satisfy wider set of requirements.
Fairness in terms of disk time vs size of IO
---------------------------------------------
An higher level controller will most likely be limited to providing fairness in terms of size/number of IO done and will find it hard to provide fairness in terms of disk time used (as CFQ provides between various prio levels). This is because only IO scheduler knows how much disk time a queue has used and information about queues and disk time used is not exported to higher layers.Quoted 2 lines
So a seeky application will still run away with lot of disk time and bring down the overall throughput of the the disk.
But that's only true if the thing is poorly implemented.
A high-level controller will need some view of the busyness of the underlying device(s). That could be "proportion of idle time", or "average length of queue" or "average request latency" or some mix of these or something else altogether.
But these things are simple to calculate, and are simple to feed back to the higher-level controller and probably don't require any changes to to IO scheduler at all, which is a great advantage.
Quoted 6 lines
And I must say that high-level throttling based upon feedback from lower layers seems like a much better model to me than hacking away in the IO scheduler layer. Both from an implementation point of view and from a "we can get it to work on things other than block devices" point of view.
Hi Andrew,
Few thoughts.
- A higher level throttling approach suffers from the issue of unfair throttling. So if there are multiple tasks in the group, who do we throttle and how do we make sure that we did throttling in proportion to the prio of tasks. Andrea's IO throttling implementation suffered from these issues. I had run some tests where RT and BW tasks were getting same BW with-in group or tasks of different prio were gettting same BW.
Even if we figure a way out to do fair throttling with-in group, underlying IO scheduler might not be CFQ at all and we should not have done so.
https://lists.linux-foundation.org/pipermail/containers/2009-May/017588.html
- Higher level throttling does not know where actually IO is going in physical layer. So we might unnecessarily be throttling IO which are going to same logical device but at the end of day to different physical devices.
Agreed that some people will want that behavior, especially in the case of max bandwidth control where one does not want to give you the BW because you did not pay for it.
So higher level controller is good for max bw control but if it comes to optimal usage of resources and do control only if needed, then it probably is not the best thing.
About the feedback thing, I am not very sure. Are you saying that we will run timed groups in higher layer and take feedback from underlying IO scheduler about how much time a group consumed or something like that and not do accounting in terms of size of IO?
Quoted 11 lines
Quoted 11 lines
Currently dm-ioband provides fairness in terms of number/size of IO.
Latencies and isolation between groups
--------------------------------------
An higher level controller is generally implementing a bandwidth throttling solution where if a group exceeds either the max bandwidth or the proportional share then throttle that group.This kind of approach will probably not help in controlling latencies as it will depend on underlying IO scheduler. Consider following scenario.
Quoted 7 lines
Quoted 3 lines
Assume there are two groups. One group is running multiple sequential readers and other group has a random reader. sequential readers will get a nice 100ms slice
Do you refer to each reader within group1, or to all readers? It would be daft if each reader in group1 were to get 100ms.
All readers in the group should get 100ms each, both in IO throttling and dm-ioband solution.
Higher level solutions are not keeping track of time slices. Time slices will be allocated by CFQ which does not have any idea about grouping. Higher level controller just keeps track of size of IO done at group level and then run either a leaky bucket or token bucket algorithm.
IO throttling is a max BW controller, so it will not even care about what is happening in other group. It will just be concerned with rate of IO in one particular group and if we exceed specified limit, throttle it. So until and unless sequential reader group hits it max bw limit, it will keep sending reads down to CFQ, and CFQ will happily assign 100ms slices to readers.
dm-ioband will not try to choke the high throughput sequential reader group for the slow random reader group because that would just kill the throughput of rotational media. Every sequential reader will run for few ms and then be throttled and this goes on. Disk will soon be seek bound.
Quoted 8 lines
Quoted 3 lines
each and then a random reader from group2 will get to dispatch the request. So latency of this random reader will depend on how many sequential readers are running in other group and that is a weak isolation between groups.
And yet that is what you appear to mean.
But surely nobody would do that - the 100ms would be assigned to and distributed amongst all readers in group1?
Dividing 100ms to all the sequential readers might not be very good on rotational media as each reader runs for small time and then seek happens. This will increase number of seeks in the system. Think of 32 sequential readers in the group and then each getting less than 3ms to run.
A better way probably is to give each queue 100ms in one run of group and then switch group. Someting like following.
SR1 RR SR2 RR SR3 RR SR4 RR...
Now each sequential reader gets 100ms and disk is not seek bound at the same time random reader latency limited by number of competing groups and not by number of processes in the group. This is what IO scheduler based IO controller is effectively doing currently.
Quoted 36 lines
Quoted 7 lines
When we control things at IO scheduler level, we assign one time slice to one group and then pick next entity to run. So effectively after one time slice (max 180ms, if prio 0 sequential reader is running), random reader in other group will get to run. Hence we achieve better isolation between groups as response time of process in a differnt group is generally not dependent on number of processes running in competing group.
I don't understand why you're comparing this implementation with such an obviously dumb competing design!
Quoted 17 lines
So a higher level solution is most likely limited to only shaping bandwidth without any control on latencies.
Stacking group scheduler on top of CFQ can lead to issues
---------------------------------------------------------
IO throttling and dm-ioband both are second level controller. That is these controllers are implemented in higher layers than io schedulers. So they control the IO at higher layer based on group policies and later IO schedulers take care of dispatching these bios to disk.Implementing a second level controller has the advantage of being able to provide bandwidth control even on logical block devices in the IO stack which don't have any IO schedulers attached to these. But they can also interefere with IO scheduling policy of underlying IO scheduler and change the effective behavior. Following are some of the issues which I think should be visible in second level controller in one form or other.
Quoted 7 lines
Prio with-in group
------------------
A second level controller can potentially interefere with behavior of different prio processes with-in a group. bios are buffered at higher layer in single queue and release of bios is FIFO and not proportionate to the ioprio of the process. This can result in a particular prio level not getting fair share.
Quoted 3 lines
That's an administrator error, isn't it? Should have put the different-priority processes into different groups.
I am thinking in practice it probably will be a mix of priority in each group. For example, consider a hypothetical scenario where two students on a university server are given two cgroups of certain weights so that IO done by these students are limited in case of contention. Now these students might want to throw in a mix of priority workload in their respective cgroup. Admin would not have any idea what priority process students are running in respective cgroup.
Quoted 8 lines
Quoted 5 lines
Buffering at higher layer can delay read requests for more than slice idle period of CFQ (default 8 ms). That means, it is possible that we are waiting for a request from the queue but it is buffered at higher layer and then idle timer will fire. It means that queue will losse its share at the same time overall throughput will be impacted as we lost those 8 ms.
That sounds like a bug.
Actually this probably is a limitation of higher level controller. It most likely is sitting so high in IO stack that it has no idea what underlying IO scheduler is and what are IO scheduler's policies. So it can't keep up with IO scheduler's policies. Secondly, it might be a low weight group and tokens might not be available fast enough to release the request.
Quoted 11 lines
Quoted 10 lines
Read Vs Write
-------------
Writes can overwhelm readers hence second level controller FIFO release will run into issue here. If there is a single queue maintained then reads will suffer large latencies. If there separate queues for reads and writes then it will be hard to decide in what ratio to dispatch reads and writes as it is IO scheduler's decision to decide when and how much read/write to dispatch. This is another place where higher level controller will not be in sync with lower level io scheduler and can change the effective policies of underlying io scheduler.
Quoted 3 lines
The IO schedulers already take care of read-vs-write and already take care of preventing large writes-starve-reads latencies (or at least, they're supposed to).
True. Actually this is a limitation of higher level controller. A higher level controller will most likely implement some of kind of queuing/buffering mechanism where it will buffer requeuests when it decides to throttle the group. Now once a fair number read and requests are buffered, and if controller is ready to dispatch some requests from the group, which requests/bio should it dispatch? reads first or writes first or reads and writes in certain ratio?
In what ratio reads and writes are dispatched is the property and decision of IO scheduler. Now higher level controller will be taking this decision and change the behavior of underlying io scheduler.
Quoted 11 lines
Quoted 5 lines
CFQ IO context Issues
---------------------
Buffering at higher layer means submission of bios later with the help of a worker thread.Why?
If it's a read, we just block the userspace process.
If it's a delayed write, the IO submission already happens in a kernel thread.
Is it ok to block pdflush on group. Some low weight group might block it for long time and hence not allow flushing out other pages. Probably that's the reason pdflush used to check if underlying device is congested or not and if it is congested, we don't go ahead with submission of request. With per bdi flusher thread things will change.
I think btrfs also has some threds which don't want to block and if underlying deivce is congested, it bails out. That's the reason I implemented per group congestion interface where if a thread does not want to block, it can check whether the group IO is going in is congested or not and will it block. So for such threads, probably higher level controller shall have to implement per group congestion interface so that threads which don't want to block can check with the controller whether it has sufficient BW to let it through and not block or may be start buffering writes in group queue.
Quoted 6 lines
If it's a synchronous write, we have to block the userspace caller anyway.
Async reads might be an issue, dunno.
I think async IO is one of the reason. IIRC, Andrea Righi, implemented the policy of returning error for async IO if group did not have sufficient tokens to dispatch the async IO and expected the application to retry later. I am not sure if that is ok.
So yes, if we are not buffering any of the read requests and either blocking the caller or returning an error (async IO) than CFQ io context is not an issue.
Quoted 8 lines
Quoted 4 lines
This changes the io context information at CFQ layer which
assigns the request to submitting thread. Change of io context info again leads to issues of idle timer expiry and issue of a process not getting fair share and reduced throughput.But we already have that problem with delayed writeback, which is a huge thing - often it's the majority of IO.
For delayed writes CFQ will not anticipate so increased anticipation timer expiry is not an issue with writes. But it probably will be an issue with reads where if higher level controller decides to block next read and CFQ is anticipating on that read. I am wondering that such kind of issues must appear with all the higher level device mapper/software raid devices also. How do they handle it. May be it is more theoritical and in practice impact is not significant.
Quoted 19 lines
Quoted 19 lines
Throughput with noop, deadline and AS
---------------------------------------------
I think an higher level controller will result in reduced overall throughput (as compared to io scheduler based io controller) and more seeks with noop, deadline and AS.The reason being, that it is likely that IO with-in a group will be related and will be relatively close as compared to IO across the groups. For example, thread pool of kvm-qemu doing IO for virtual machine. In case of higher level control, IO from various groups will go into a single queue at lower level controller and it might happen that IO is now interleaved (G1, G2, G1, G3, G4....) causing more seeks and reduced throughput. (Agreed that merging will help up to some extent but still....).
Instead, in case of lower level controller, IO scheduler maintains one queue per group hence there is no interleaving of IO between groups. And if IO is related with-in group, then we shoud get reduced number/amount of seek and higher throughput.
Quoted 6 lines
Quoted 2 lines
Latency can be a concern but that can be controlled by reducing the time slice length of the queue.
Well maybe, maybe not. If a group is throttled, it isn't submitting new IO. The unthrottled group is doing the IO submitting and that IO will have decent locality.
But throttling will kick in ocassionaly. Rest of the time both the groups will be dispatching bios at the same time. So for most part of it IO scheduler will probably see IO from both the groups and there will be small intervals where one group is completely throttled and IO scheduler is busy dispatching requests only from a single group.
Quoted 31 lines
Quoted 31 lines
Fairness at logical device level vs at physical device level
------------------------------------------------------------IO scheduler based controller has the limitation that it works only with the bottom most devices in the IO stack where IO scheduler is attached.
For example, assume a user has created a logical device lv0 using three underlying disks sda, sdb and sdc. Also assume there are two tasks T1 and T2 in two groups doing IO on lv0. Also assume that weights of groups are in the ratio of 2:1 so T1 should get double the BW of T2 on lv0 device.
T1 T2 \ / lv0 / | \ sda sdb sdcNow resource control will take place only on devices sda, sdb and sdc and not at lv0 level. So if IO from two tasks is relatively uniformly distributed across the disks then T1 and T2 will see the throughput ratio in proportion to weight specified. But if IO from T1 and T2 is going to different disks and there is no contention then at higher level they both will see same BW.
Here a second level controller can produce better fairness numbers at logical device but most likely at redued overall throughput of the system, because it will try to control IO even if there is no contention at phsical possibly leaving diksks unused in the system.
Quoted 7 lines
Quoted 5 lines
Hence, question comes that how important it is to control bandwidth at higher level logical devices also. The actual contention for resources is at the leaf block device so it probably makes sense to do any kind of control there and not at the intermediate devices. Secondly probably it also means better use of available resources.
hm. What will be the effects of this limitation in real-world use?
In some cases user/application will not see the bandwidth ratio between two groups in same proportion as assigned weights and primary reason for that will be that this workload did not create enough contention for physical resources unerneath.
So it all depends on what kind of bandwidth gurantees are we offering. If we are saying that we provide good fairness numbers at logical devices irrespective of whether resources are not used optimally, then it will be irritating for the user.
I think it also might become an issue once we implement max bandwidth control. We will not be able to define max bandwidth on a logical device and an application will get more than max bandwidth if it is doing IO to different underlying devices.
I would say that leaf node control is good for optimal resource usage and for proportional BW control, but not a good fit for max bandwidth control.
Quoted 41 lines
Quoted 10 lines
Limited Fairness
----------------
Currently CFQ idles on a sequential reader queue to make sure it gets its fair share. A second level controller will find it tricky to anticipate. Either it will not have any anticipation logic and in that case it will not provide fairness to single readers in a group (as dm-ioband does) or if it starts anticipating then we should run into these strange situations where second level controller is anticipating on one queue/group and underlying IO scheduler might be anticipating on something else.It depends on the size of the inter-group timeslices. If the amount of time for which a group is unthrottled is "large" comapred to the typical anticipation times, this issue fades away.
And those timeslices _should_ be large. Because as you mentioned above, different groups are probably working different parts of the disk.
Quoted 22 lines
Need of device mapper tools
---------------------------
A device mapper based solution will require creation of a ioband device on each physical/logical device one wants to control. So it requires usage of device mapper tools even for the people who are not using device mapper. At the same time creation of ioband device on each partition in the system to control the IO can be cumbersome and overwhelming if system has got lots of disks and partitions with-in.IMHO, IO scheduler based IO controller is a reasonable approach to solve the problem of group bandwidth control, and can do hierarchical IO scheduling more tightly and efficiently.
But I am all ears to alternative approaches and suggestions how doing things can be done better and will be glad to implement it.
TODO
====
- code cleanups, testing, bug fixing, optimizations, benchmarking etc...
- More testing to make sure there are no regressions in CFQ.
Quoted 8 lines
Quoted 6 lines
Testing
=======Environment
==========
A 7200 RPM SATA drive with queue depth of 31. Ext3 filesystem.That's a bit of a toy.
Yes it is. :-)
Quoted 3 lines
Do we have testing results for more enterprisey hardware? Big storage arrays? SSD? Infiniband? iscsi? nfs? (lol, gotcha)
Not yet. I will try to get hold of some storage arrays and run some tests.
Quoted 52 lines
Quoted 29 lines
I am mostly
running fio jobs which have been limited to 30 seconds run and then monitored the throughput and latency.Test1: Random Reader Vs Random Writers
======================================
Launched a random reader and then increasing number of random writers to see the effect on random reader BW and max lantecies.[fio --rw=randwrite --bs=64K --size=2G --runtime=30 --direct=1 --ioengine=libaio --iodepth=4 --numjobs= <1 to 32> ] [fio --rw=randread --bs=4K --size=2G --runtime=30 --direct=1]
[Vanilla CFQ, No groups]
<--------------random writers--------------------> <------random reader-->nr Max-bdwidth Min-bdwidth Agg-bdwidth Max-latency Agg-bdwidth Max-latency 1 5737KiB/s 5737KiB/s 5737KiB/s 164K usec 503KiB/s 159K usec 2 2055KiB/s 1984KiB/s 4039KiB/s 1459K usec 150KiB/s 170K usec 4 1238KiB/s 932KiB/s 4419KiB/s 4332K usec 153KiB/s 225K usec 8 1059KiB/s 929KiB/s 7901KiB/s 1260K usec 118KiB/s 377K usec 16 604KiB/s 483KiB/s 8519KiB/s 3081K usec 47KiB/s 756K usec 32 367KiB/s 222KiB/s 9643KiB/s 5940K usec 22KiB/s 923K usecCreated two cgroups group1 and group2 of weights 500 each. Launched increasing number of random writers in group1 and one random reader in group2 using fio.
[IO controller CFQ; group_idle=8; group1 weight=500; group2 weight=500] <--------------random writers(group1)-------------> <-random reader(group2)->
Quoted 7 lines
nr Max-bdwidth Min-bdwidth Agg-bdwidth Max-latency Agg-bdwidth Max-latency 1 18115KiB/s 18115KiB/s 18115KiB/s 604K usec 345KiB/s 176K usec 2 3752KiB/s 3676KiB/s 7427KiB/s 4367K usec 402KiB/s 187K usec 4 1951KiB/s 1863KiB/s 7642KiB/s 1989K usec 384KiB/s 181K usec 8 755KiB/s 629KiB/s 5683KiB/s 2133K usec 366KiB/s 319K usec 16 418KiB/s 369KiB/s 6276KiB/s 1323K usec 352KiB/s 287K usec 32 236KiB/s 191KiB/s 6518KiB/s 1910K usec 337KiB/s 273K usecThat's a good result.
Quoted 13 lines
Also ran the same test with IO controller CFQ in flat mode to see if there are any major deviations from Vanilla CFQ. Does not look like any.
[IO controller CFQ; No groups ]
<--------------random writers--------------------> <------random reader-->nr Max-bdwidth Min-bdwidth Agg-bdwidth Max-latency Agg-bdwidth Max-latency 1 5696KiB/s 5696KiB/s 5696KiB/s 259K usec 500KiB/s 194K usec 2 2483KiB/s 2197KiB/s 4680KiB/s 887K usec 150KiB/s 159K usec 4 1471KiB/s 1433KiB/s 5817KiB/s 962K usec 126KiB/s 189K usec 8 691KiB/s 580KiB/s 5159KiB/s 2752K usec 197KiB/s 246K usec 16 781KiB/s 698KiB/s 11892KiB/s 943K usec 61KiB/s 529K usec 32 415KiB/s 324KiB/s 12461KiB/s 4614K usec 17KiB/s 737K usec
Quoted 7 lines
Quoted 3 lines
Notes:
- With vanilla CFQ, random writers can overwhelm a random reader. Bring down its throughput and bump up latencies significantly.Isn't that a CFQ shortcoming which we should address separately? If so, the comparisons aren't presently valid because we're comparing with a CFQ which has known, should-be-fixed problems.
I am not sure if it is a CFQ issue. These are synchronous random writes. These are equally important as random reader. So now CFQ has 33 synchronous queues to serve. Becuase it does not know about groups, it has no choice but to serve them in round robin manner. So it does not sound like a CFQ issue. Just that CFQ can give random reader an advantage if it knows that random reader is in a different group and that's where IO controller comes in to picture.
Quoted 129 lines
Quoted 110 lines
- With IO controller, one can provide isolation to the random reader group and maintain consitent view of bandwidth and latencies.
Test2: Random Reader Vs Sequential Reader
========================================
Launched a random reader and then increasing number of sequential readers to see the effect on BW and latencies of random reader.[fio --rw=read --bs=4K --size=2G --runtime=30 --direct=1 --numjobs= <1 to 16> ] [fio --rw=randread --bs=4K --size=2G --runtime=30 --direct=1]
[ Vanilla CFQ, No groups ]
<---------------seq readers----------------------> <------random reader-->nr Max-bdwidth Min-bdwidth Agg-bdwidth Max-latency Agg-bdwidth Max-latency 1 23318KiB/s 23318KiB/s 23318KiB/s 55940 usec 36KiB/s 247K usec 2 14732KiB/s 11406KiB/s 26126KiB/s 142K usec 20KiB/s 446K usec 4 9417KiB/s 5169KiB/s 27338KiB/s 404K usec 10KiB/s 993K usec 8 3360KiB/s 3041KiB/s 25850KiB/s 954K usec 60KiB/s 956K usec 16 1888KiB/s 1457KiB/s 26763KiB/s 1871K usec 28KiB/s 1868K usecCreated two cgroups group1 and group2 of weights 500 each. Launched increasing number of sequential readers in group1 and one random reader in group2 using fio.
[IO controller CFQ; group_idle=1; group1 weight=500; group2 weight=500] <---------------group1---------------------------> <------group2--------->
nr Max-bdwidth Min-bdwidth Agg-bdwidth Max-latency Agg-bdwidth Max-latency 1 13733KiB/s 13733KiB/s 13733KiB/s 247K usec 330KiB/s 154K usec 2 8553KiB/s 4963KiB/s 13514KiB/s 472K usec 322KiB/s 174K usec 4 5045KiB/s 1367KiB/s 13134KiB/s 947K usec 318KiB/s 178K usec 8 1774KiB/s 1420KiB/s 13035KiB/s 1871K usec 323KiB/s 233K usec 16 959KiB/s 518KiB/s 12691KiB/s 3809K usec 324KiB/s 208K usecAlso ran the same test with IO controller CFQ in flat mode to see if there are any major deviations from Vanilla CFQ. Does not look like any.
[IO controller CFQ; No groups ]
<---------------seq readers----------------------> <------random reader-->nr Max-bdwidth Min-bdwidth Agg-bdwidth Max-latency Agg-bdwidth Max-latency 1 23028KiB/s 23028KiB/s 23028KiB/s 47460 usec 36KiB/s 253K usec 2 14452KiB/s 11176KiB/s 25628KiB/s 145K usec 20KiB/s 447K usec 4 8815KiB/s 5720KiB/s 27121KiB/s 396K usec 10KiB/s 968K usec 8 3335KiB/s 2827KiB/s 24866KiB/s 960K usec 62KiB/s 955K usec 16 1784KiB/s 1311KiB/s 26537KiB/s 1883K usec 26KiB/s 1866K usecNotes:
- The BW and latencies of random reader in group 2 seems to be stable and bounded and does not get impacted much as number of sequential readers increase in group1. Hence provding good isolation.- Throughput of sequential readers comes down and latencies go up as half of disk bandwidth (in terms of time) has been reserved for random reader group.
Test3: Sequential Reader Vs Sequential Reader
============================================
Created two cgroups group1 and group2 of weights 500 and 1000 respectively. Launched increasing number of sequential readers in group1 and one sequential reader in group2 using fio and monitored how bandwidth is being distributed between two groups.First 5 columns give stats about job in group1 and last two columns give stats about job in group2.
<---------------group1---------------------------> <------group2--------->
nr Max-bdwidth Min-bdwidth Agg-bdwidth Max-latency Agg-bdwidth Max-latency 1 8970KiB/s 8970KiB/s 8970KiB/s 230K usec 20681KiB/s 124K usec 2 6783KiB/s 3202KiB/s 9984KiB/s 546K usec 19682KiB/s 139K usec 4 4641KiB/s 1029KiB/s 9280KiB/s 1185K usec 19235KiB/s 172K usec 8 1435KiB/s 1079KiB/s 9926KiB/s 2461K usec 19501KiB/s 153K usec 16 764KiB/s 398KiB/s 9395KiB/s 4986K usec 19367KiB/s 172K usecNote: group2 is getting double the bandwidth of group1 even in the face of increasing number of readers in group1.
Test4 (Isolation between two KVM virtual machines)
==================================================
Created two KVM virtual machines. Partitioned a disk on host in two partitions and gave one partition to each virtual machine. Put both the virtual machines in two different cgroup of weight 1000 and 500 each. Virtual machines created ext3 file system on the partitions exported from host and did buffered writes. Host seems writes as synchronous and virtual machine with higher weight gets double the disk time of virtual machine of lower weight. Used deadline scheduler in this test case.Some more details about configuration are in documentation patch.
Test5 (Fairness for async writes, Buffered Write Vs Buffered Write)
===================================================================
Fairness for async writes is tricky and biggest reason is that async writes are cached in higher layers (page cahe) as well as possibly in file system layer also (btrfs, xfs etc), and are dispatched to lower layers not necessarily in proportional manner.For example, consider two dd threads reading
/dev/zeroas input file and doing writes of huge files. Very soon we will crossvm_dirty_ratioand dd thread will be forced to write out some pages to disk before more pages can be dirtied. But not necessarily dirty pages of same thread are picked. It can very well pick the inode of lesser priority dd thread and do some writeout. So effectively higher weight dd is doing writeouts of lower weight dd pages and we don't see service differentation.IOW, the core problem with buffered write fairness is that higher weight thread does not throw enought IO traffic at IO controller to keep the queue continuously backlogged. In my testing, there are many .2 to .8 second intervals where higher weight queue is empty and in that duration lower weight queue get lots of job done giving the impression that there was no service differentiation.
Quoted 5 lines
In summary, from IO controller point of view async writes support is there. Because page cache has not been designed in such a manner that higher prio/weight writer can do more write out as compared to lower prio/weight writer, gettting service differentiation is hard and it is visible in some cases and not visible in some cases.
Here's where it all falls to pieces.
For async writeback we just don't care about IO priorities. Because from the point of view of the userspace task, the write was async! It occurred at memory bandwidth speed.
It's only when the kernel's dirty memory thresholds start to get exceeded that we start to care about prioritisation. And at that time, all dirty memory (within a memcg?) is equal - a high-ioprio dirty page consumes just as much memory as a low-ioprio dirty page.
So when
balance_dirty_pages()hits, what do we want to do?
Quoted 7 lines
I suppose that all we can do is to block low-ioprio processes more agressively at the VFS layer, to reduce the rate at which they're dirtying memory so as to give high-ioprio processes more of the disk bandwidth.
But you've gone and implemented all of this stuff at the io-controller level and not at the VFS level so you're, umm, screwed.
True that's an issue. For async writes we don't create parallel IO paths from user space to IO scheduler hence it is hard to provide fairness in all the cases. I think part of the problem is page cache and some serialization also comes from kjournald.
How about coming up with another cgroup controller for buffered writes or clubbing it with memory controller as KAMEZAWA Hiroyuki suggested and co-mount this with io controller? This should help control buffered writes per cgroup.
Quoted 119 lines
Importantly screwed! It's a very common workload pattern, and one which causes tremendous amounts of IO to be generated very quickly, traditionally causing bad latency effects all over the place. And we have no answer to this.
Quoted 113 lines
Vanilla CFQ Vs IO Controller CFQ
================================
We have not fundamentally changed CFQ, instead enhanced it to also support hierarchical io scheduling. In the process invariably there are small changes here and there as new scenarios come up. Running some tests here and comparing both the CFQ's to see if there is any major deviation in behavior.Test1: Sequential Readers
=========================
[fio --rw=read --bs=4K --size=2G --runtime=30 --direct=1 --numjobs=<1 to 16> ]IO scheduler: Vanilla CFQ
nr Max-bdwidth Min-bdwidth Agg-bdwidth Max-latency 1 35499KiB/s 35499KiB/s 35499KiB/s 19195 usec 2 17089KiB/s 13600KiB/s 30690KiB/s 118K usec 4 9165KiB/s 5421KiB/s 29411KiB/s 380K usec 8 3815KiB/s 3423KiB/s 29312KiB/s 830K usec 16 1911KiB/s 1554KiB/s 28921KiB/s 1756K usecIO scheduler: IO controller CFQ
nr Max-bdwidth Min-bdwidth Agg-bdwidth Max-latency 1 34494KiB/s 34494KiB/s 34494KiB/s 14482 usec 2 16983KiB/s 13632KiB/s 30616KiB/s 123K usec 4 9237KiB/s 5809KiB/s 29631KiB/s 372K usec 8 3901KiB/s 3505KiB/s 29162KiB/s 822K usec 16 1895KiB/s 1653KiB/s 28945KiB/s 1778K usecTest2: Sequential Writers
=========================
[fio --rw=write --bs=4K --size=2G --runtime=30 --direct=1 --numjobs=<1 to 16> ]IO scheduler: Vanilla CFQ
nr Max-bdwidth Min-bdwidth Agg-bdwidth Max-latency 1 22669KiB/s 22669KiB/s 22669KiB/s 401K usec 2 14760KiB/s 7419KiB/s 22179KiB/s 571K usec 4 5862KiB/s 5746KiB/s 23174KiB/s 444K usec 8 3377KiB/s 2199KiB/s 22427KiB/s 1057K usec 16 2229KiB/s 556KiB/s 20601KiB/s 5099K usecIO scheduler: IO Controller CFQ
nr Max-bdwidth Min-bdwidth Agg-bdwidth Max-latency 1 22911KiB/s 22911KiB/s 22911KiB/s 37319 usec 2 11752KiB/s 11632KiB/s 23383KiB/s 245K usec 4 6663KiB/s 5409KiB/s 23207KiB/s 384K usec 8 3161KiB/s 2460KiB/s 22566KiB/s 935K usec 16 1888KiB/s 795KiB/s 21349KiB/s 3009K usecTest3: Random Readers
=========================
[fio --rw=randread --bs=4K --size=2G --runtime=30 --direct=1 --numjobs=1 to 16]IO scheduler: Vanilla CFQ
nr Max-bdwidth Min-bdwidth Agg-bdwidth Max-latency 1 484KiB/s 484KiB/s 484KiB/s 22596 usec 2 229KiB/s 196KiB/s 425KiB/s 51111 usec 4 119KiB/s 73KiB/s 405KiB/s 2344 msec 8 93KiB/s 23KiB/s 399KiB/s 2246 msec 16 38KiB/s 8KiB/s 328KiB/s 3965 msecIO scheduler: IO Controller CFQ
nr Max-bdwidth Min-bdwidth Agg-bdwidth Max-latency 1 483KiB/s 483KiB/s 483KiB/s 29391 usec 2 229KiB/s 196KiB/s 426KiB/s 51625 usec 4 132KiB/s 88KiB/s 417KiB/s 2313 msec 8 79KiB/s 18KiB/s 389KiB/s 2298 msec 16 43KiB/s 9KiB/s 327KiB/s 3905 msecTest4: Random Writers
=====================
[fio --rw=randwrite --bs=4K --size=2G --runtime=30 --direct=1 --numjobs=1 to 16]IO scheduler: Vanilla CFQ
nr Max-bdwidth Min-bdwidth Agg-bdwidth Max-latency 1 14641KiB/s 14641KiB/s 14641KiB/s 93045 usec 2 7896KiB/s 1348KiB/s 9245KiB/s 82778 usec 4 2657KiB/s 265KiB/s 6025KiB/s 216K usec 8 951KiB/s 122KiB/s 3386KiB/s 1148K usec 16 66KiB/s 22KiB/s 829KiB/s 1308 msecIO scheduler: IO Controller CFQ
nr Max-bdwidth Min-bdwidth Agg-bdwidth Max-latency 1 14454KiB/s 14454KiB/s 14454KiB/s 74623 usec 2 4595KiB/s 4104KiB/s 8699KiB/s 135K usec 4 3113KiB/s 334KiB/s 5782KiB/s 200K usec 8 1146KiB/s 95KiB/s 3832KiB/s 593K usec 16 71KiB/s 29KiB/s 814KiB/s 1457 msecNotes:
- Does not look like that anything has changed significantly.Previous versions of the patches were posted here.
------------------------------------------------(V1) http://lkml.org/lkml/2009/3/11/486
(V2) http://lkml.org/lkml/2009/5/5/275
(V3) http://lkml.org/lkml/2009/5/26/472
(V4) http://lkml.org/lkml/2009/6/8/580
(V5) http://lkml.org/lkml/2009/6/19/279
(V6) http://lkml.org/lkml/2009/7/2/369
(V7) http://lkml.org/lkml/2009/7/24/253
(V8) http://lkml.org/lkml/2009/8/16/204
(V9) http://lkml.org/lkml/2009/8/28/327Thanks
Vivek
More performance numbers (Was: Re: IO scheduler based IO controller V10)
In reply to a message not in the archiveOn Thu, Sep 24, 2009 at 02:33:15PM -0700, Andrew Morton wrote:
[..]
Quoted 4 lines
Quoted 4 lines
Testing
=======
Quoted 10 lines
Quoted 3 lines
Environment
==========
A 7200 RPM SATA drive with queue depth of 31. Ext3 filesystem.That's a bit of a toy.
Do we have testing results for more enterprisey hardware? Big storage arrays? SSD? Infiniband? iscsi? nfs? (lol, gotcha)
Hi Andrew,
I got hold of a relatively more enterprisey stuff. It is an storage array with few striped disks(I think 4 or 5). So this is not high end stuff but better than my single SATA disk. I guess may be entry level enterprisy stuff. Still trying to get hold of higher end configuration...
Apart from IO scheduler controller number, I also got a chance to run same tests with dm-ioband controller. I am posting these too. I am also planning to run similar numbers on Andrea's "max bw" controller also. Should be able to post those numbers also in 2-3 days.
Software Environment
====================
- 2.6.31 kernel
- V10 of IO scheduler based controller
- version v1.14.0 of dm-ioband patches
Used fio jobs for 30 seconds in various configurations. All the IO is direct IO to eliminate the effects of caches.
I have run three sets for each test. Blindly reporting results of set2 from each test, otherwise it is too much of data to report.
Had lun of 2500GB capacity. Used 200G partitions with ext3 file system for my testing. For IO scheduler based controller patches, created two cgroups of weight 100 each doing IO to single 200G partition.
For dm-ioband, created two partitions of 200G each and created two ioband devices of weight 100 each with policy "weight-iosize". Ideally I should haved used cgroups on dm-ioband also but could not get cgroup patch going. Because this is striped configuration, not expecting any major changes in results due to that.
Sequential reader vs Random reader
==================================
Launched on random reader in one group and launched increasing number of sequential readers in other group to see the effect on latency and bandwidth of random reader.
[fio1 --rw=read --bs=4K --size=2G --runtime=30 --direct=1 ]
[fio2 --rw=randread --bs=4K --size=1G --runtime=30 --direct=1 --group_reporting]
Vanilla CFQ
-----------
[Sequential readers] [Random Reader]
nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 13806KB/s 13806KB/s 13483KB/s 28672 usec 1 23KB/s 212 msec 2 6406KB/s 6268KB/s 12378KB/s 128K usec 1 10KB/s 453 msec 4 3934KB/s 2536KB/s 13103KB/s 321K usec 1 6KB/s 847 msec 8 1934KB/s 556KB/s 13009KB/s 876K usec 1 13KB/s 1632 msec 16 958KB/s 280KB/s 13761KB/s 1621K usec 1 10KB/s 3217 msec 32 512KB/s 126KB/s 13861KB/s 3241K usec 1 6KB/s 3249 msec
IO scheduler controller + CFQ
-----------------------------
[Sequential readers] [Random Reader]
nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 5651KB/s 5651KB/s 5519KB/s 126K usec 1 222KB/s 130K usec 2 3144KB/s 1479KB/s 4515KB/s 347K usec 1 225KB/s 189K usec 4 1852KB/s 626KB/s 5128KB/s 775K usec 1 224KB/s 159K usec 8 971KB/s 279KB/s 6464KB/s 1666K usec 1 222KB/s 193K usec 16 454KB/s 129KB/s 6293KB/s 3356K usec 1 218KB/s 466K usec 32 239KB/s 42KB/s 5986KB/s 6753K usec 1 214KB/s 503K usec
Notes:
- The BW and latency of random reader are fairly stable in the face of increasing number of sequential readers. There are couple of spikes in latency, i guess comes from the hardware somehow. But will debug more to make sure that I am not delaying in dispatch of request.
dm-ioaband + CFQ
----------------
[Sequential readers] [Random Reader]
nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 12466KB/s 12466KB/s 12174KB/s 40078 usec 1 37KB/s 221 msec 2 6240KB/s 5904KB/s 11859KB/s 134K usec 1 12KB/s 443 msec 4 3517KB/s 2529KB/s 12368KB/s 357K usec 1 6KB/s 772 msec 8 1779KB/s 594KB/s 9857KB/s 719K usec 1 60KB/s 852K usec 16 914KB/s 300KB/s 10934KB/s 1467K usec 1 40KB/s 1285K usec 32 589KB/s 187KB/s 11537KB/s 3547K usec 1 14KB/s 3228 msec
Notes:
- Does not look like we provide fairness to random reader here. Latencies are on the rise and BW is on the decline. this is almost like Vanilla CFQ with reduced overall throughput.
- dm-ioband claims that they do not provide fairness for slow moving group and I think it is a bad idea. This leads to very weak isolation with no benefits. Especially if a buffered writer is running in other group. This should be fixed.
Random writers vs Random reader
================================
[fio1 --rw=randwrite --bs=64K --size=2G --runtime=30 --ioengine=libaio --iodepth=4 --direct=1 ] [fio2 --rw=randread --bs=4K --size=1G --runtime=30 --direct=1 --group_reporting]
Vanilla CFQ
-----------
[Random Writers] [Random Reader]
nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 67785KB/s 67785KB/s 66197KB/s 45499 usec 1 170KB/s 94098 usec 2 35163KB/s 35163KB/s 68678KB/s 218K usec 1 75KB/s 2335 msec 4 17759KB/s 15308KB/s 64206KB/s 2387K usec 1 85KB/s 2331 msec 8 8725KB/s 6495KB/s 57120KB/s 3761K usec 1 67KB/s 2488K usec 16 3912KB/s 3456KB/s 57121KB/s 1273K usec 1 60KB/s 1668K usec 32 2020KB/s 1503KB/s 56786KB/s 4221K usec 1 39KB/s 1101 msec
IO scheduler controller + CFQ
-----------------------------
[Random Writers] [Random Reader]
nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 20919KB/s 20919KB/s 20428KB/s 288K usec 1 213KB/s 580K usec 2 14765KB/s 14674KB/s 28749KB/s 776K usec 1 203KB/s 112K usec 4 7177KB/s 7091KB/s 27839KB/s 970K usec 1 197KB/s 132K usec 8 3027KB/s 2953KB/s 23285KB/s 3145K usec 1 218KB/s 203K usec 16 1959KB/s 1750KB/s 28919KB/s 1266K usec 1 160KB/s 182K usec 32 908KB/s 753KB/s 26267KB/s 2091K usec 1 208KB/s 144K usec
Notes:
- Again disk time has been divided half and half between random reader group and random writer group. Fairly stable BW and latencies for random reader in the face of increasing number of random writers.
- Drop in aggregate bw of random writers is expected as they now get only half of disk time.
dm-ioaband + CFQ
----------------
[Random Writers] [Random Reader]
nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 63659KB/s 63659KB/s 62167KB/s 89954 usec 1 164KB/s 72 msec 2 27109KB/s 27096KB/s 52933KB/s 674K usec 1 140KB/s 2204K usec 4 16553KB/s 16216KB/s 63946KB/s 694K usec 1 56KB/s 1871 msec 8 3907KB/s 3347KB/s 28752KB/s 2406K usec 1 226KB/s 2407K usec 16 2841KB/s 2647KB/s 42334KB/s 870K usec 1 52KB/s 3043 msec 32 738KB/s 657KB/s 21285KB/s 1529K usec 1 21KB/s 4435 msec
Notes:
- Again no fairness for random reader. Decreasing BW, increasing latency. No isolation in this case.
- I am curious what happened to random writer throughput in case of "32" writers. We did not get higher BW for random reader but random writer still suffering in throughput for random writer. I can see this for all the three sets.
Sequential Readers vs Sequential reader
=======================================
[fio1 --rw=read --bs=4K --size=2G --runtime=30 --direct=1] [fio2 --rw=read --bs=4K --size=2G --runtime=30 --direct=1]
Vanilla CFQ
-----------
[Sequential Readers] [Sequential Reader]
nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 6434KB/s 6434KB/s 6283KB/s 107K usec 1 7017KB/s 111K usec 2 4688KB/s 3284KB/s 7785KB/s 274K usec 1 4541KB/s 218K usec 4 3365KB/s 1326KB/s 9769KB/s 597K usec 1 3038KB/s 424K usec 8 1827KB/s 504KB/s 12053KB/s 813K usec 1 1389KB/s 813K usec 16 1022KB/s 301KB/s 13954KB/s 1618K usec 1 676KB/s 1617K usec 32 494KB/s 149KB/s 13611KB/s 3216K usec 1 416KB/s 3215K usec
IO scheduler controller + CFQ
-----------------------------
[Sequential Readers] [Sequential Reader]
nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 6605KB/s 6605KB/s 6450KB/s 120K usec 1 6527KB/s 120K usec 2 3706KB/s 1985KB/s 5558KB/s 323K usec 1 6331KB/s 149K usec 4 2053KB/s 672KB/s 5731KB/s 721K usec 1 6267KB/s 148K usec 8 1013KB/s 337KB/s 6962KB/s 1525K usec 1 6136KB/s 120K usec 16 497KB/s 125KB/s 6873KB/s 3226K usec 1 5882KB/s 113K usec 32 297KB/s 48KB/s 6445KB/s 6394K usec 1 5767KB/s 116K usec
Notes:
- Stable BW and lateneis for sequential reader in the face of increasing number of readers in other group.
dm-ioaband + CFQ
----------------
[Sequential Readers] [Sequential Reader]
nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 7140KB/s 7140KB/s 6972KB/s 112K usec 1 6886KB/s 165K usec 2 3965KB/s 2762KB/s 6569KB/s 479K usec 1 5887KB/s 475K usec 4 2725KB/s 1483KB/s 7999KB/s 532K usec 1 4774KB/s 500K usec 8 1610KB/s 621KB/s 9565KB/s 729K usec 1 2910KB/s 677K usec 16 904KB/s 319KB/s 10809KB/s 1431K usec 1 1970KB/s 1399K usec 32 553KB/s 8KB/s 11794KB/s 2330K usec 1 1337KB/s 2398K usec
Notes:
- Decreasing throughput and increasing latencies for sequential reader. Hence no isolation in this case.
- Also note the in case of "32" readers, difference between "max-bw" and "min-bw" is relatively large, considering that all the 32 readers are of same prio. So bw distribution with-in group is not very good. This is the issue of ioprio with-in group I have pointed many times. Ryo is looking into it now.
Sequential Readers vs Multiple Random Readers
=======================================
Ok, because dm-ioband does not provide fairness in case if heavy IO activity is not going in the group, I decided to run a slightly different test case where 16 sequential readers are running in one group and I run increasing number of random readers in other group to see when do I start getting fairness and its effect.
[fio1 --rw=read --bs=4K --size=2G --runtime=30 --direct=1 ]
[fio2 --rw=randread --bs=4K --size=1G --runtime=30 --direct=1 --group_reporting]
Vanilla CFQ
-----------
[Sequential Readers] [Multiple Random Readers]
nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 16 961KB/s 280KB/s 13978KB/s 1673K usec 1 10KB/s 3223 msec 16 903KB/s 260KB/s 12925KB/s 1770K usec 2 28KB/s 3465 msec 16 832KB/s 231KB/s 11428KB/s 2088K usec 4 57KB/s 3891K usec 16 765KB/s 187KB/s 9899KB/s 2500K usec 8 99KB/s 3937K usec 16 512KB/s 144KB/s 6759KB/s 3451K usec 16 148KB/s 5470K usec
IO scheduler controller + CFQ
-----------------------------
[Sequential Readers] [Multiple Random Readers]
nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 16 456KB/s 112KB/s 6380KB/s 3361K usec 1 221KB/s 503K usec 16 476KB/s 159KB/s 6040KB/s 3432K usec 2 214KB/s 549K usec 16 606KB/s 178KB/s 6052KB/s 3801K usec 4 177KB/s 1341K usec 16 589KB/s 83KB/s 6243KB/s 3394K usec 8 154KB/s 3288K usec 16 547KB/s 122KB/s 6122KB/s 3538K usec 16 145KB/s 5959K usec
Notes:
- Stable BW and latencies for sequential reader group in the face of increasing number of random readers in other group.
- Because disk is divided half/half in terms of time, random reader group also gets decent amount of job done. Not sure why BW dips a bit when number of random readers increases. Too seeky to handle?
dm-ioaband + CFQ
----------------
[Sequential Readers] [Multiple Random Readers]
nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 16 926KB/s 293KB/s 10256KB/s 1634K usec 1 55KB/s 1377K usec 16 906KB/s 284KB/s 9240KB/s 1825K usec 2 71KB/s 2392K usec 16 321KB/s 18KB/s 1621KB/s 2037K usec 4 326KB/s 2054K usec 16 188KB/s 16KB/s 1188KB/s 9757K usec 8 404KB/s 3269K usec 16 167KB/s 64KB/s 1700KB/s 2859K usec 16 1064KB/s 2920K usec
Notes:
- Looks like ioband tried to provide fairness from the time when number of random readers are 4. Note, there is sudden increase in BW of random readers and drastic drop in BW of sequential readers.
- By the time number of readers reach 16, total array throughput reduces to around 2.7 MB/s. It got killed because suddenly we are trying to provide fairness in terms of size of IO. That's why on seeky media fairness in terms of disk time works better.
- There is no isolation between groups. Throughput of sequential reader group continues to drop and latencies rise.
- I think these are serious issues which should be looked into and fixed.
Thanks
Vivek
Performance numbers with IO throttling patches (Was: Re: IO scheduler based IO controller V10)
In reply to a message not in the archiveOn Thu, Sep 24, 2009 at 02:33:15PM -0700, Andrew Morton wrote:
[..]
Quoted 10 lines
Quoted 3 lines
Environment
==========
A 7200 RPM SATA drive with queue depth of 31. Ext3 filesystem.That's a bit of a toy.
Do we have testing results for more enterprisey hardware? Big storage arrays? SSD? Infiniband? iscsi? nfs? (lol, gotcha)
Hi All,
Couple of days back I posted some performance number of "IO scheduler controller" and "dm-ioband" here.
http://lkml.org/lkml/2009/10/8/9
Now I have run similar tests with Andrea Righi's IO throttling approach of max bandwidth control. This is the exercise to understand pros/cons of each approach and see how can we take things forward.
Environment
===========
Software
--------
- 2.6.31 kenrel
- IO scheduler controller V10 on top of 2.6.31
- IO throttling patch on top of 2.6.31. Patch is available here.
http://www.develer.com/~arighi/linux/patches/io-throttle/old/cgroup-io-throttle-2.6.31.patch
Hardware
--------
A storage array of 5 striped disks of 500GB each.
Used fio jobs for 30 seconds in various configurations. Most of the IO is direct IO to eliminate the effects of caches.
I have run three sets for each test. Blindly reporting results of set2 from each test, otherwise it is too much of data to report.
Had lun of 2500GB capacity. Used 200G partition with ext3 file system for my testing. For IO scheduler controller testing, created two cgroups of weight 100 each so that effectively disk can be divided half/half between two groups.
For IO throttling patches also created two cgroups. Now tricky part is that it is a max bw controller and not a proportional weight controller. So dividing the disk capacity half/half between two cgroups is tricky. The reason being I just don't know what's the BW capacity of underlying storage. Throughput varies so much with type of workload. For example, on my arrary, this is how throughput looks like with different workloads.
8 sequential buffered readers 115 MB/s
8 direct sequential readers bs=64K 64 MB/s
8 direct sequential readers bs=4K 14 MB/s
8 buffered random readers bs=64K 3 MB/s
8 direct random readers bs=64K 15 MB/s
8 direct random readers bs=4K 1.5 MB/s
So throughput seems to be varying from 1.5 MB/s to 115 MB/s depending on workload. What should be the BW limits per cgroup to divide disk BW in half/half between two groups?
So I took a conservative estimate and divide max bandwidth divide by 2, and thought of array capacity as 60MB/s and assign each cgroup 30MB/s. In some cases I have assigened even 10MB/s or 5MB/s to each cgropu to see the effects of throttling. I am using "Leaky bucket" policy for all the tests.
As theme of two controllers is different, at some places it might sound like apples vs oranges comparison. But still it does help...
Multiple Random Reader vs Sequential Reader
===============================================
Generally random readers bring the throughput down of others in the system. Ran a test to see the impact of increasing number of random readers on single sequential reader in different groups.
Vanilla CFQ
-----------------------------------
[Multiple Random Reader] [Sequential Reader]
nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 23KB/s 23KB/s 22KB/s 691 msec 1 13519KB/s 468K usec 2 152KB/s 152KB/s 297KB/s 244K usec 1 12380KB/s 31675 usec 4 174KB/s 156KB/s 638KB/s 249K usec 1 10860KB/s 36715 usec 8 49KB/s 11KB/s 310KB/s 1856 msec 1 1292KB/s 990K usec 16 63KB/s 48KB/s 877KB/s 762K usec 1 3905KB/s 506K usec 32 35KB/s 27KB/s 951KB/s 2655 msec 1 1109KB/s 1910K usec
IO scheduler controller + CFQ
-----------------------------------
[Multiple Random Reader] [Sequential Reader]
nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 228KB/s 228KB/s 223KB/s 132K usec 1 5551KB/s 129K usec 2 97KB/s 97KB/s 190KB/s 154K usec 1 5718KB/s 122K usec 4 115KB/s 110KB/s 445KB/s 208K usec 1 5909KB/s 116K usec 8 23KB/s 12KB/s 158KB/s 2820 msec 1 5445KB/s 168K usec 16 11KB/s 3KB/s 145KB/s 5963 msec 1 5418KB/s 164K usec 32 6KB/s 2KB/s 139KB/s 12762 msec 1 5398KB/s 175K usec
Notes:
- Sequential reader in group2 seems to be well isolated from random readers in group1. Throughput and latency of sequential reader are stable and don't drop as number of random readers inrease in system.
io-throttle + CFQ
------------------
BW limit group1=10 MB/s BW limit group2=10 MB/s
[Multiple Random Reader] [Sequential Reader]
nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 37KB/s 37KB/s 36KB/s 218K usec 1 8006KB/s 20529 usec 2 185KB/s 183KB/s 360KB/s 228K usec 1 7475KB/s 33665 usec 4 188KB/s 171KB/s 699KB/s 262K usec 1 6800KB/s 46224 usec 8 84KB/s 51KB/s 573KB/s 1800K usec 1 2835KB/s 885K usec 16 21KB/s 9KB/s 294KB/s 3590 msec 1 437KB/s 1855K usec 32 34KB/s 27KB/s 980KB/s 2861K usec 1 1145KB/s 1952K usec
Notes:
- I have setup limits of 10MB/s in both the cgroups. Now random reader group will never achieve that kind of speed, so it will not be throttled and then it goes onto impact the throughput and latency of other groups in the system.
- Now the key question is how conservative one should in be setting up max BW limit. On this box if a customer has bought 10MB/s cgroup and if he is running some random readers it will kill throughput of other groups in the system and their latencies will shoot up. No isolation in this case.
- So in general, max BW provides isolation from high speed groups but it does not provide isolaton from random reader groups which are moving slow.
Multiple Sequential Reader vs Random Reader
===============================================
Now running a reverse test where in one group I am running increasing number of sequential readers and in other group I am running one random reader and see the impact of sequential readers on random reader.
Vanilla CFQ
-----------------------------------
[Multiple Sequential Reader] [Random Reader]
nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 13978KB/s 13978KB/s 13650KB/s 27614 usec 1 22KB/s 227 msec 2 6225KB/s 6166KB/s 12101KB/s 568K usec 1 10KB/s 457 msec 4 4052KB/s 2462KB/s 13107KB/s 322K usec 1 6KB/s 841 msec 8 1899KB/s 557KB/s 12960KB/s 829K usec 1 13KB/s 1628 msec 16 1007KB/s 279KB/s 13833KB/s 1629K usec 1 10KB/s 3236 msec 32 506KB/s 98KB/s 13704KB/s 3389K usec 1 6KB/s 3238 msec
IO scheduler controller + CFQ
-----------------------------------
[Multiple Sequential Reader] [Random Reader]
nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 5721KB/s 5721KB/s 5587KB/s 126K usec 1 223KB/s 126K usec 2 3216KB/s 1442KB/s 4549KB/s 349K usec 1 224KB/s 176K usec 4 1895KB/s 640KB/s 5121KB/s 775K usec 1 222KB/s 189K usec 8 957KB/s 285KB/s 6368KB/s 1680K usec 1 223KB/s 142K usec 16 458KB/s 132KB/s 6455KB/s 3343K usec 1 219KB/s 165K usec 32 248KB/s 55KB/s 6001KB/s 6957K usec 1 220KB/s 504K usec
Notes:
- Random reader is well isolated from increasing number of sequential readers in other group. BW and latencies are stable.
io-throttle + CFQ
-----------------------------------
BW limit group1=10 MB/s BW limit group2=10 MB/s
[Multiple Sequential Reader] [Random Reader]
nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 8200KB/s 8200KB/s 8007KB/s 20275 usec 1 37KB/s 217K usec 2 3926KB/s 3919KB/s 7661KB/s 122K usec 1 16KB/s 441 msec 4 2271KB/s 1497KB/s 7672KB/s 611K usec 1 9KB/s 927 msec 8 1113KB/s 513KB/s 7507KB/s 849K usec 1 21KB/s 1020 msec 16 661KB/s 236KB/s 7959KB/s 1679K usec 1 13KB/s 2926 msec 32 292KB/s 109KB/s 7864KB/s 3446K usec 1 8KB/s 3439 msec
BW limit group1=5 MB/s BW limit group2=5 MB/s
[Multiple Sequential Reader] [Random Reader]
nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 4686KB/s 4686KB/s 4576KB/s 21095 usec 1 57KB/s 219K usec 2 2298KB/s 2179KB/s 4372KB/s 132K usec 1 37KB/s 431K usec 4 1245KB/s 1019KB/s 4449KB/s 324K usec 1 26KB/s 835 msec 8 584KB/s 403KB/s 4109KB/s 833K usec 1 30KB/s 1625K usec 16 346KB/s 252KB/s 4605KB/s 1641K usec 1 129KB/s 3236K usec 32 175KB/s 56KB/s 4269KB/s 3236K usec 1 8KB/s 3235 msec
Notes:
- Above result is surprising to me. I have run it twice. In first run, I setup per cgroup limit as 10MB/s and in second run I set it up 5MB/s. In both the cases as number of sequential readers increase in other groups, random reader's throughput decreases and latencies increase. This is happening despite the fact that sequential readers are being throttled to make sure it does not impact workload in other group. Wondering why random readers are not seeing consistent throughput and latencies.
- Andrea, can you please also run similar tests to see if you see same results or not. This is to rule out any testing methodology errors or scripting bugs. :-). I also have collected the snapshot of some cgroup files like bandwidth-max, throttlecnt, and stats. Let me know if you want those to see what is happenig here.
Multiple Sequential Reader vs Sequential Reader
===============================================
- This time running random readers are out of the picture and trying to see the effect of increasing number of sequential readers on another sequential reader running in a different group.
Vanilla CFQ
-----------------------------------
[Multiple Sequential Reader] [Sequential Reader]
nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 6325KB/s 6325KB/s 6176KB/s 114K usec 1 6902KB/s 120K usec 2 4588KB/s 3102KB/s 7510KB/s 571K usec 1 4564KB/s 680K usec 4 3242KB/s 1158KB/s 9469KB/s 495K usec 1 3198KB/s 410K usec 8 1775KB/s 459KB/s 12011KB/s 1178K usec 1 1366KB/s 818K usec 16 943KB/s 296KB/s 13285KB/s 1923K usec 1 728KB/s 1816K usec 32 511KB/s 148KB/s 13555KB/s 3286K usec 1 391KB/s 3212K usec
IO scheduler controller + CFQ
-----------------------------------
[Multiple Sequential Reader] [Sequential Reader]
nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 6781KB/s 6781KB/s 6622KB/s 109K usec 1 6691KB/s 115K usec 2 3758KB/s 1876KB/s 5502KB/s 693K usec 1 6373KB/s 419K usec 4 2100KB/s 671KB/s 5751KB/s 987K usec 1 6330KB/s 569K usec 8 1023KB/s 355KB/s 6969KB/s 1569K usec 1 6086KB/s 120K usec 16 520KB/s 130KB/s 7094KB/s 3140K usec 1 5984KB/s 119K usec 32 245KB/s 86KB/s 6621KB/s 6571K usec 1 5850KB/s 113K usec
Notes:
- BW and latencies of sequential reader in group 2 are fairly stable as number of readers increase in first group.
io-throttle + CFQ
-----------------------------------
BW limit group1=30 MB/s BW limit group2=30 MB/s
[Multiple Sequential Reader] [Sequential Reader]
nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 6343KB/s 6343KB/s 6195KB/s 116K usec 1 6993KB/s 109K usec 2 4583KB/s 3046KB/s 7451KB/s 583K usec 1 4516KB/s 433K usec 4 2945KB/s 1324KB/s 9552KB/s 602K usec 1 3001KB/s 583K usec 8 1804KB/s 473KB/s 12257KB/s 861K usec 1 1386KB/s 815K usec 16 942KB/s 265KB/s 13560KB/s 1659K usec 1 718KB/s 1658K usec 32 462KB/s 143KB/s 13757KB/s 3482K usec 1 409KB/s 3480K usec
Notes:
- BW decreases and latencies increase in group2 as number of readers increase in first group. This should be due to fact that no throttling will happen as none of the groups is hitting the limit of 30MB/s. To me this is the tricky part. How a service provider is supposed to set the limit of groups. If groups are not hitting max limits, it will still impact the BW and latencies in other group.
BW limit group1=10 MB/s BW limit group2=10 MB/s
[Multiple Sequential Reader] [Sequential Reader]
nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 4128KB/s 4128KB/s 4032KB/s 215K usec 1 4076KB/s 170K usec 2 2880KB/s 1886KB/s 4655KB/s 291K usec 1 2891KB/s 212K usec 4 1912KB/s 888KB/s 5872KB/s 417K usec 1 1881KB/s 411K usec 8 1032KB/s 432KB/s 7312KB/s 841K usec 1 853KB/s 816K usec 16 540KB/s 259KB/s 7844KB/s 1728K usec 1 503KB/s 1609K usec 32 291KB/s 111KB/s 7920KB/s 3417K usec 1 249KB/s 3205K usec
Notes:
- Same test with 10MB/s as group limit. This is again a surprising result. Max BW in first group is being throttled but still throughput is dropping significantly in second group and latencies are on the rise.
- Limit of first group is 10MB/s but it is achieving max BW of around 8MB/s only. What happened to rest of the 2MB/s?
- Andrea, again, please do run this test. The throughput drop in second group stumps me and forces me to think if I am doing something wrong.
BW limit group1=5 MB/s BW limit group2=5 MB/s
[Multiple Sequential Reader] [Sequential Reader]
nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 2434KB/s 2434KB/s 2377KB/s 110K usec 1 2415KB/s 120K usec 2 1639KB/s 1186KB/s 2759KB/s 222K usec 1 1709KB/s 220K usec 4 1114KB/s 648KB/s 3314KB/s 420K usec 1 1163KB/s 414K usec 8 567KB/s 366KB/s 4060KB/s 901K usec 1 527KB/s 816K usec 16 329KB/s 179KB/s 4324KB/s 1613K usec 1 311KB/s 1613K usec 32 178KB/s 70KB/s 4320KB/s 3235K usec 1 163KB/s 3209K usec
- Setting the limit to 5MB/s per group also does not seem to be helping the second group.
Multiple Random Writer vs Random Reader
===============================================
This time running multiple random writers in first group and see the impact on throughput and latency of random reader in different group.
Vanilla CFQ
-----------------------------------
[Multiple Random Writer] [Random Reader]
nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 64018KB/s 64018KB/s 62517KB/s 353K usec 1 190KB/s 96 msec 2 35298KB/s 35257KB/s 68899KB/s 208K usec 1 76KB/s 2416 msec 4 16387KB/s 14662KB/s 60630KB/s 3746K usec 1 106KB/s 2308K usec 8 5106KB/s 3492KB/s 33335KB/s 2995K usec 1 193KB/s 2292K usec 16 3676KB/s 3002KB/s 51807KB/s 2283K usec 1 72KB/s 2298K usec 32 2169KB/s 1480KB/s 56882KB/s 1990K usec 1 35KB/s 1093 msec
IO scheduler controller + CFQ
-----------------------------------
[Multiple Random Writer] [Random Reader]
nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 20369KB/s 20369KB/s 19892KB/s 877K usec 1 255KB/s 137K usec 2 14347KB/s 14288KB/s 27964KB/s 1010K usec 1 228KB/s 117K usec 4 6996KB/s 6701KB/s 26775KB/s 1362K usec 1 221KB/s 180K usec 8 2849KB/s 2770KB/s 22007KB/s 2660K usec 1 250KB/s 485K usec 16 1463KB/s 1365KB/s 22384KB/s 2606K usec 1 254KB/s 115K usec 32 799KB/s 681KB/s 22404KB/s 2879K usec 1 266KB/s 107K usec
Notes
- BW and latencies of random reader in second group are fairly stable.
io-throttle + CFQ
-----------------------------------
BW limit group1=30 MB/s BW limit group2=30 MB/s
[Multiple Random Writer] [Random Reader]
nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 21920KB/s 21920KB/s 21406KB/s 1017K usec 1 353KB/s 432K usec 2 14291KB/s 9626KB/s 23357KB/s 1832K usec 1 362KB/s 177K usec 4 7130KB/s 5135KB/s 24736KB/s 1336K usec 1 348KB/s 425K usec 8 3165KB/s 2949KB/s 23792KB/s 2133K usec 1 336KB/s 146K usec 16 1653KB/s 1406KB/s 23694KB/s 2198K usec 1 337KB/s 115K usec 32 793KB/s 717KB/s 23198KB/s 2195K usec 1 330KB/s 192K usec
BW limit group1=10 MB/s BW limit group2=10 MB/s
[Multiple Random Writer] [Random Reader]
nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 7903KB/s 7903KB/s 7718KB/s 1037K usec 1 474KB/s 103K usec 2 4496KB/s 4428KB/s 8715KB/s 1091K usec 1 450KB/s 553K usec 4 2153KB/s 1827KB/s 7914KB/s 2042K usec 1 458KB/s 108K usec 8 1129KB/s 1087KB/s 8688KB/s 1280K usec 1 432KB/s 98215 usec 16 606KB/s 527KB/s 8668KB/s 2303K usec 1 426KB/s 90609 usec 32 312KB/s 259KB/s 8599KB/s 2557K usec 1 441KB/s 95283 usec
Notes:
- IO throttling seems to be working really well here. Random writers are contained in the first group and this gives stable BW and latencies to random reader in second group.
Multiple Buffered Writer vs Buffered Writer
===========================================
This time run multiple buffered writers in group1 and see run a single buffered writer in other group and see if we can provide fairness and isolation.
Vanilla CFQ
------------
[Multiple Buffered Writer] [Buffered Writer]
nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 68997KB/s 68997KB/s 67380KB/s 645K usec 1 67122KB/s 567K usec 2 47509KB/s 46218KB/s 91510KB/s 865K usec 1 45118KB/s 865K usec 4 28002KB/s 26906KB/s 105MB/s 1649K usec 1 26879KB/s 1643K usec 8 15985KB/s 14849KB/s 117MB/s 943K usec 1 15653KB/s 766K usec 16 11567KB/s 6881KB/s 128MB/s 1174K usec 1 7333KB/s 947K usec 32 5877KB/s 3649KB/s 130MB/s 1205K usec 1 5142KB/s 988K usec
IO scheduler controller + CFQ
-----------------------------------
[Multiple Buffered Writer] [Buffered Writer]
nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 68580KB/s 68580KB/s 66972KB/s 2901K usec 1 67194KB/s 2901K usec 2 47419KB/s 45700KB/s 90936KB/s 3149K usec 1 44628KB/s 2377K usec 4 27825KB/s 27274KB/s 105MB/s 1177K usec 1 27584KB/s 1177K usec 8 15382KB/s 14288KB/s 114MB/s 1539K usec 1 14794KB/s 783K usec 16 9161KB/s 7592KB/s 124MB/s 3177K usec 1 7713KB/s 886K usec 32 4928KB/s 3961KB/s 126MB/s 1152K usec 1 6465KB/s 4510K usec
Notes:
- It does not work. Buffered writer in second group are being overwhelmed by writers in group1.
- This is a limitation of IO scheduler based controller currently as page cache at higher layer evens out the traffic and does not throw more traffic from higher weight group.
- This is something needs more work at higher layers like dirty limts per cgroup in memory contoller and the method to writeout buffered pages belonging to a particular memory cgroup. This is still being brainstormed.
io-throttle + CFQ
-----------------------------------
BW limit group1=30 MB/s BW limit group2=30 MB/s
[Multiple Buffered Writer] [Buffered Writer]
nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 33863KB/s 33863KB/s 33070KB/s 3046K usec 1 25165KB/s 13248K usec 2 13457KB/s 12906KB/s 25745KB/s 9286K usec 1 29958KB/s 3736K usec 4 7414KB/s 6543KB/s 27145KB/s 10557K usec 1 30968KB/s 8356K usec 8 3562KB/s 2640KB/s 24430KB/s 12012K usec 1 30801KB/s 7037K usec 16 3962KB/s 881KB/s 26632KB/s 12650K usec 1 31150KB/s 7173K usec 32 3275KB/s 406KB/s 27295KB/s 14609K usec 1 26328KB/s 8069K usec
Notes:
- This seems to work well here. io-throttle is throttling the writers before they write too much of data in page cache. One side effect of this seems to be that now a process will not be allowed to write at memory speed in page cahce and will be limited to disk IO speed limits set for the cgroup.
Andrea is thinking of removing throttling in balance_dirty_pages() to allow writting at disk speed till we hit dirty_limits. But removing it leads to a different issue where too many dirty pages from a single group can be present from a cgroup in page cache and if that cgroup is slow moving one, then pages are flushed to disk at slower speed delyaing other higher rate cgroups. (all discussed in private mails with Andrea).
ioprio class and iopriority with-in cgroups issues with IO-throttle
===================================================================
Currently throttling logic is designed in such a way that it makes the throttling uniform for every process in the group. So we will loose the differentiation between different class of processes or differnetitation between different priority of processes with-in group.
I have run the tests of these in the past and reported it here in the past.
https://lists.linux-foundation.org/pipermail/containers/2009-May/017588.html
Thanks
Vivek
Quoted 104 lines · Andrew Morton
Quoted 5 lines · On Thu, 24 Sep 2009 15:25:04 -0400 Vivek Goyal <[email protected]> wrote:
Hi All,
Here is the V10 of the IO controller patches generated on top of 2.6.31.
Thanks for the writeup. It really helps and is most worthwhile for a project of this importance, size and complexity.
Quoted 76 lines
What problem are we trying to solve
===================================
Provide group IO scheduling feature in Linux along the lines of other resource controllers like cpu.IOW, provide facility so that a user can group applications using cgroups and control the amount of disk time/bandwidth received by a group based on its weight.
How to solve the problem
=========================Different people have solved the issue differetnly. So far looks it looks like we seem to have following two core requirements when it comes to fairness at group level.
- Control bandwidth seen by groups.
- Control on latencies when a request gets backlogged in group.At least there are now three patchsets available (including this one).
IO throttling
-------------
This is a bandwidth controller which keeps track of IO rate of a group and throttles the process in the group if it exceeds the user specified limit.dm-ioband
---------
This is a proportional bandwidth controller implemented as device mapper driver and provides fair access in terms of amount of IO done (not in terms of disk time as CFQ does).So one will setup one or more dm-ioband devices on top of physical/logical block device, configure the ioband device and pass information like grouping etc. Now this device will keep track of bios flowing through it and control the flow of bios based on group policies.
IO scheduler based IO controller
--------------------------------
Here we have viewed the problem of IO contoller as hierarchical group scheduling (along the lines of CFS group scheduling) issue. Currently one can view linux IO schedulers as flat where there is one root group and all the IO belongs to that group.This patchset basically modifies IO schedulers to also support hierarchical group scheduling. CFQ already provides fairness among different processes. I have extended it support group IO schduling. Also took some of the code out of CFQ and put in a common layer so that same group scheduling code can be used by noop, deadline and AS to support group scheduling.
Pros/Cons
=========
There are pros and cons to each of the approach. Following are some of the thoughts.Max bandwidth vs proportional bandwidth
---------------------------------------
IO throttling is a max bandwidth controller and not a proportional one. Additionaly it provides fairness in terms of amount of IO done (and not in terms of disk time as CFQ does).Personally, I think that proportional weight controller is useful to more people than just max bandwidth controller. In addition, IO scheduler based controller can also be enhanced to do max bandwidth control. So it can satisfy wider set of requirements.
Fairness in terms of disk time vs size of IO
---------------------------------------------
An higher level controller will most likely be limited to providing fairness in terms of size/number of IO done and will find it hard to provide fairness in terms of disk time used (as CFQ provides between various prio levels). This is because only IO scheduler knows how much disk time a queue has used and information about queues and disk time used is not exported to higher layers.Quoted 2 lines
So a seeky application will still run away with lot of disk time and bring down the overall throughput of the the disk.
But that's only true if the thing is poorly implemented.
A high-level controller will need some view of the busyness of the underlying device(s). That could be "proportion of idle time", or "average length of queue" or "average request latency" or some mix of these or something else altogether.
But these things are simple to calculate, and are simple to feed back to the higher-level controller and probably don't require any changes to to IO scheduler at all, which is a great advantage.
Quoted 6 lines
And I must say that high-level throttling based upon feedback from lower layers seems like a much better model to me than hacking away in the IO scheduler layer. Both from an implementation point of view and from a "we can get it to work on things other than block devices" point of view.
Hi Andrew,
Few thoughts.
- A higher level throttling approach suffers from the issue of unfair throttling. So if there are multiple tasks in the group, who do we throttle and how do we make sure that we did throttling in proportion to the prio of tasks. Andrea's IO throttling implementation suffered from these issues. I had run some tests where RT and BW tasks were getting same BW with-in group or tasks of different prio were gettting same BW.
Even if we figure a way out to do fair throttling with-in group, underlying IO scheduler might not be CFQ at all and we should not have done so.
https://lists.linux-foundation.org/pipermail/containers/2009-May/017588.html
- Higher level throttling does not know where actually IO is going in physical layer. So we might unnecessarily be throttling IO which are going to same logical device but at the end of day to different physical devices.
Agreed that some people will want that behavior, especially in the case of max bandwidth control where one does not want to give you the BW because you did not pay for it.
So higher level controller is good for max bw control but if it comes to optimal usage of resources and do control only if needed, then it probably is not the best thing.
About the feedback thing, I am not very sure. Are you saying that we will run timed groups in higher layer and take feedback from underlying IO scheduler about how much time a group consumed or something like that and not do accounting in terms of size of IO?
Quoted 11 lines
Quoted 11 lines
Currently dm-ioband provides fairness in terms of number/size of IO.
Latencies and isolation between groups
--------------------------------------
An higher level controller is generally implementing a bandwidth throttling solution where if a group exceeds either the max bandwidth or the proportional share then throttle that group.This kind of approach will probably not help in controlling latencies as it will depend on underlying IO scheduler. Consider following scenario.
Quoted 7 lines
Quoted 3 lines
Assume there are two groups. One group is running multiple sequential readers and other group has a random reader. sequential readers will get a nice 100ms slice
Do you refer to each reader within group1, or to all readers? It would be daft if each reader in group1 were to get 100ms.
All readers in the group should get 100ms each, both in IO throttling and dm-ioband solution.
Higher level solutions are not keeping track of time slices. Time slices will be allocated by CFQ which does not have any idea about grouping. Higher level controller just keeps track of size of IO done at group level and then run either a leaky bucket or token bucket algorithm.
IO throttling is a max BW controller, so it will not even care about what is happening in other group. It will just be concerned with rate of IO in one particular group and if we exceed specified limit, throttle it. So until and unless sequential reader group hits it max bw limit, it will keep sending reads down to CFQ, and CFQ will happily assign 100ms slices to readers.
dm-ioband will not try to choke the high throughput sequential reader group for the slow random reader group because that would just kill the throughput of rotational media. Every sequential reader will run for few ms and then be throttled and this goes on. Disk will soon be seek bound.
Quoted 8 lines
Quoted 3 lines
each and then a random reader from group2 will get to dispatch the request. So latency of this random reader will depend on how many sequential readers are running in other group and that is a weak isolation between groups.
And yet that is what you appear to mean.
But surely nobody would do that - the 100ms would be assigned to and distributed amongst all readers in group1?
Dividing 100ms to all the sequential readers might not be very good on rotational media as each reader runs for small time and then seek happens. This will increase number of seeks in the system. Think of 32 sequential readers in the group and then each getting less than 3ms to run.
A better way probably is to give each queue 100ms in one run of group and then switch group. Someting like following.
SR1 RR SR2 RR SR3 RR SR4 RR...
Now each sequential reader gets 100ms and disk is not seek bound at the same time random reader latency limited by number of competing groups and not by number of processes in the group. This is what IO scheduler based IO controller is effectively doing currently.
Quoted 36 lines
Quoted 7 lines
When we control things at IO scheduler level, we assign one time slice to one group and then pick next entity to run. So effectively after one time slice (max 180ms, if prio 0 sequential reader is running), random reader in other group will get to run. Hence we achieve better isolation between groups as response time of process in a differnt group is generally not dependent on number of processes running in competing group.
I don't understand why you're comparing this implementation with such an obviously dumb competing design!
Quoted 17 lines
So a higher level solution is most likely limited to only shaping bandwidth without any control on latencies.
Stacking group scheduler on top of CFQ can lead to issues
---------------------------------------------------------
IO throttling and dm-ioband both are second level controller. That is these controllers are implemented in higher layers than io schedulers. So they control the IO at higher layer based on group policies and later IO schedulers take care of dispatching these bios to disk.Implementing a second level controller has the advantage of being able to provide bandwidth control even on logical block devices in the IO stack which don't have any IO schedulers attached to these. But they can also interefere with IO scheduling policy of underlying IO scheduler and change the effective behavior. Following are some of the issues which I think should be visible in second level controller in one form or other.
Quoted 7 lines
Prio with-in group
------------------
A second level controller can potentially interefere with behavior of different prio processes with-in a group. bios are buffered at higher layer in single queue and release of bios is FIFO and not proportionate to the ioprio of the process. This can result in a particular prio level not getting fair share.
Quoted 3 lines
That's an administrator error, isn't it? Should have put the different-priority processes into different groups.
I am thinking in practice it probably will be a mix of priority in each group. For example, consider a hypothetical scenario where two students on a university server are given two cgroups of certain weights so that IO done by these students are limited in case of contention. Now these students might want to throw in a mix of priority workload in their respective cgroup. Admin would not have any idea what priority process students are running in respective cgroup.
Quoted 8 lines
Quoted 5 lines
Buffering at higher layer can delay read requests for more than slice idle period of CFQ (default 8 ms). That means, it is possible that we are waiting for a request from the queue but it is buffered at higher layer and then idle timer will fire. It means that queue will losse its share at the same time overall throughput will be impacted as we lost those 8 ms.
That sounds like a bug.
Actually this probably is a limitation of higher level controller. It most likely is sitting so high in IO stack that it has no idea what underlying IO scheduler is and what are IO scheduler's policies. So it can't keep up with IO scheduler's policies. Secondly, it might be a low weight group and tokens might not be available fast enough to release the request.
Quoted 11 lines
Quoted 10 lines
Read Vs Write
-------------
Writes can overwhelm readers hence second level controller FIFO release will run into issue here. If there is a single queue maintained then reads will suffer large latencies. If there separate queues for reads and writes then it will be hard to decide in what ratio to dispatch reads and writes as it is IO scheduler's decision to decide when and how much read/write to dispatch. This is another place where higher level controller will not be in sync with lower level io scheduler and can change the effective policies of underlying io scheduler.
Quoted 3 lines
The IO schedulers already take care of read-vs-write and already take care of preventing large writes-starve-reads latencies (or at least, they're supposed to).
True. Actually this is a limitation of higher level controller. A higher level controller will most likely implement some of kind of queuing/buffering mechanism where it will buffer requeuests when it decides to throttle the group. Now once a fair number read and requests are buffered, and if controller is ready to dispatch some requests from the group, which requests/bio should it dispatch? reads first or writes first or reads and writes in certain ratio?
In what ratio reads and writes are dispatched is the property and decision of IO scheduler. Now higher level controller will be taking this decision and change the behavior of underlying io scheduler.
Quoted 11 lines
Quoted 5 lines
CFQ IO context Issues
---------------------
Buffering at higher layer means submission of bios later with the help of a worker thread.Why?
If it's a read, we just block the userspace process.
If it's a delayed write, the IO submission already happens in a kernel thread.
Is it ok to block pdflush on group. Some low weight group might block it for long time and hence not allow flushing out other pages. Probably that's the reason pdflush used to check if underlying device is congested or not and if it is congested, we don't go ahead with submission of request. With per bdi flusher thread things will change.
I think btrfs also has some threds which don't want to block and if underlying deivce is congested, it bails out. That's the reason I implemented per group congestion interface where if a thread does not want to block, it can check whether the group IO is going in is congested or not and will it block. So for such threads, probably higher level controller shall have to implement per group congestion interface so that threads which don't want to block can check with the controller whether it has sufficient BW to let it through and not block or may be start buffering writes in group queue.
Quoted 6 lines
If it's a synchronous write, we have to block the userspace caller anyway.
Async reads might be an issue, dunno.
I think async IO is one of the reason. IIRC, Andrea Righi, implemented the policy of returning error for async IO if group did not have sufficient tokens to dispatch the async IO and expected the application to retry later. I am not sure if that is ok.
So yes, if we are not buffering any of the read requests and either blocking the caller or returning an error (async IO) than CFQ io context is not an issue.
Quoted 8 lines
Quoted 4 lines
This changes the io context information at CFQ layer which
assigns the request to submitting thread. Change of io context info again leads to issues of idle timer expiry and issue of a process not getting fair share and reduced throughput.But we already have that problem with delayed writeback, which is a huge thing - often it's the majority of IO.
For delayed writes CFQ will not anticipate so increased anticipation timer expiry is not an issue with writes. But it probably will be an issue with reads where if higher level controller decides to block next read and CFQ is anticipating on that read. I am wondering that such kind of issues must appear with all the higher level device mapper/software raid devices also. How do they handle it. May be it is more theoritical and in practice impact is not significant.
Quoted 19 lines
Quoted 19 lines
Throughput with noop, deadline and AS
---------------------------------------------
I think an higher level controller will result in reduced overall throughput (as compared to io scheduler based io controller) and more seeks with noop, deadline and AS.The reason being, that it is likely that IO with-in a group will be related and will be relatively close as compared to IO across the groups. For example, thread pool of kvm-qemu doing IO for virtual machine. In case of higher level control, IO from various groups will go into a single queue at lower level controller and it might happen that IO is now interleaved (G1, G2, G1, G3, G4....) causing more seeks and reduced throughput. (Agreed that merging will help up to some extent but still....).
Instead, in case of lower level controller, IO scheduler maintains one queue per group hence there is no interleaving of IO between groups. And if IO is related with-in group, then we shoud get reduced number/amount of seek and higher throughput.
Quoted 6 lines
Quoted 2 lines
Latency can be a concern but that can be controlled by reducing the time slice length of the queue.
Well maybe, maybe not. If a group is throttled, it isn't submitting new IO. The unthrottled group is doing the IO submitting and that IO will have decent locality.
But throttling will kick in ocassionaly. Rest of the time both the groups will be dispatching bios at the same time. So for most part of it IO scheduler will probably see IO from both the groups and there will be small intervals where one group is completely throttled and IO scheduler is busy dispatching requests only from a single group.
Quoted 31 lines
Quoted 31 lines
Fairness at logical device level vs at physical device level
------------------------------------------------------------IO scheduler based controller has the limitation that it works only with the bottom most devices in the IO stack where IO scheduler is attached.
For example, assume a user has created a logical device lv0 using three underlying disks sda, sdb and sdc. Also assume there are two tasks T1 and T2 in two groups doing IO on lv0. Also assume that weights of groups are in the ratio of 2:1 so T1 should get double the BW of T2 on lv0 device.
T1 T2 \ / lv0 / | \ sda sdb sdcNow resource control will take place only on devices sda, sdb and sdc and not at lv0 level. So if IO from two tasks is relatively uniformly distributed across the disks then T1 and T2 will see the throughput ratio in proportion to weight specified. But if IO from T1 and T2 is going to different disks and there is no contention then at higher level they both will see same BW.
Here a second level controller can produce better fairness numbers at logical device but most likely at redued overall throughput of the system, because it will try to control IO even if there is no contention at phsical possibly leaving diksks unused in the system.
Quoted 7 lines
Quoted 5 lines
Hence, question comes that how important it is to control bandwidth at higher level logical devices also. The actual contention for resources is at the leaf block device so it probably makes sense to do any kind of control there and not at the intermediate devices. Secondly probably it also means better use of available resources.
hm. What will be the effects of this limitation in real-world use?
In some cases user/application will not see the bandwidth ratio between two groups in same proportion as assigned weights and primary reason for that will be that this workload did not create enough contention for physical resources unerneath.
So it all depends on what kind of bandwidth gurantees are we offering. If we are saying that we provide good fairness numbers at logical devices irrespective of whether resources are not used optimally, then it will be irritating for the user.
I think it also might become an issue once we implement max bandwidth control. We will not be able to define max bandwidth on a logical device and an application will get more than max bandwidth if it is doing IO to different underlying devices.
I would say that leaf node control is good for optimal resource usage and for proportional BW control, but not a good fit for max bandwidth control.
Quoted 41 lines
Quoted 10 lines
Limited Fairness
----------------
Currently CFQ idles on a sequential reader queue to make sure it gets its fair share. A second level controller will find it tricky to anticipate. Either it will not have any anticipation logic and in that case it will not provide fairness to single readers in a group (as dm-ioband does) or if it starts anticipating then we should run into these strange situations where second level controller is anticipating on one queue/group and underlying IO scheduler might be anticipating on something else.It depends on the size of the inter-group timeslices. If the amount of time for which a group is unthrottled is "large" comapred to the typical anticipation times, this issue fades away.
And those timeslices _should_ be large. Because as you mentioned above, different groups are probably working different parts of the disk.
Quoted 22 lines
Need of device mapper tools
---------------------------
A device mapper based solution will require creation of a ioband device on each physical/logical device one wants to control. So it requires usage of device mapper tools even for the people who are not using device mapper. At the same time creation of ioband device on each partition in the system to control the IO can be cumbersome and overwhelming if system has got lots of disks and partitions with-in.IMHO, IO scheduler based IO controller is a reasonable approach to solve the problem of group bandwidth control, and can do hierarchical IO scheduling more tightly and efficiently.
But I am all ears to alternative approaches and suggestions how doing things can be done better and will be glad to implement it.
TODO
====
- code cleanups, testing, bug fixing, optimizations, benchmarking etc...
- More testing to make sure there are no regressions in CFQ.
Quoted 8 lines
Quoted 6 lines
Testing
=======Environment
==========
A 7200 RPM SATA drive with queue depth of 31. Ext3 filesystem.That's a bit of a toy.
Yes it is. :-)
Quoted 3 lines
Do we have testing results for more enterprisey hardware? Big storage arrays? SSD? Infiniband? iscsi? nfs? (lol, gotcha)
Not yet. I will try to get hold of some storage arrays and run some tests.
Quoted 52 lines
Quoted 29 lines
I am mostly
running fio jobs which have been limited to 30 seconds run and then monitored the throughput and latency.Test1: Random Reader Vs Random Writers
======================================
Launched a random reader and then increasing number of random writers to see the effect on random reader BW and max lantecies.[fio --rw=randwrite --bs=64K --size=2G --runtime=30 --direct=1 --ioengine=libaio --iodepth=4 --numjobs= <1 to 32> ] [fio --rw=randread --bs=4K --size=2G --runtime=30 --direct=1]
[Vanilla CFQ, No groups]
<--------------random writers--------------------> <------random reader-->nr Max-bdwidth Min-bdwidth Agg-bdwidth Max-latency Agg-bdwidth Max-latency 1 5737KiB/s 5737KiB/s 5737KiB/s 164K usec 503KiB/s 159K usec 2 2055KiB/s 1984KiB/s 4039KiB/s 1459K usec 150KiB/s 170K usec 4 1238KiB/s 932KiB/s 4419KiB/s 4332K usec 153KiB/s 225K usec 8 1059KiB/s 929KiB/s 7901KiB/s 1260K usec 118KiB/s 377K usec 16 604KiB/s 483KiB/s 8519KiB/s 3081K usec 47KiB/s 756K usec 32 367KiB/s 222KiB/s 9643KiB/s 5940K usec 22KiB/s 923K usecCreated two cgroups group1 and group2 of weights 500 each. Launched increasing number of random writers in group1 and one random reader in group2 using fio.
[IO controller CFQ; group_idle=8; group1 weight=500; group2 weight=500] <--------------random writers(group1)-------------> <-random reader(group2)->
Quoted 7 lines
nr Max-bdwidth Min-bdwidth Agg-bdwidth Max-latency Agg-bdwidth Max-latency 1 18115KiB/s 18115KiB/s 18115KiB/s 604K usec 345KiB/s 176K usec 2 3752KiB/s 3676KiB/s 7427KiB/s 4367K usec 402KiB/s 187K usec 4 1951KiB/s 1863KiB/s 7642KiB/s 1989K usec 384KiB/s 181K usec 8 755KiB/s 629KiB/s 5683KiB/s 2133K usec 366KiB/s 319K usec 16 418KiB/s 369KiB/s 6276KiB/s 1323K usec 352KiB/s 287K usec 32 236KiB/s 191KiB/s 6518KiB/s 1910K usec 337KiB/s 273K usecThat's a good result.
Quoted 13 lines
Also ran the same test with IO controller CFQ in flat mode to see if there are any major deviations from Vanilla CFQ. Does not look like any.
[IO controller CFQ; No groups ]
<--------------random writers--------------------> <------random reader-->nr Max-bdwidth Min-bdwidth Agg-bdwidth Max-latency Agg-bdwidth Max-latency 1 5696KiB/s 5696KiB/s 5696KiB/s 259K usec 500KiB/s 194K usec 2 2483KiB/s 2197KiB/s 4680KiB/s 887K usec 150KiB/s 159K usec 4 1471KiB/s 1433KiB/s 5817KiB/s 962K usec 126KiB/s 189K usec 8 691KiB/s 580KiB/s 5159KiB/s 2752K usec 197KiB/s 246K usec 16 781KiB/s 698KiB/s 11892KiB/s 943K usec 61KiB/s 529K usec 32 415KiB/s 324KiB/s 12461KiB/s 4614K usec 17KiB/s 737K usec
Quoted 7 lines
Quoted 3 lines
Notes:
- With vanilla CFQ, random writers can overwhelm a random reader. Bring down its throughput and bump up latencies significantly.Isn't that a CFQ shortcoming which we should address separately? If so, the comparisons aren't presently valid because we're comparing with a CFQ which has known, should-be-fixed problems.
I am not sure if it is a CFQ issue. These are synchronous random writes. These are equally important as random reader. So now CFQ has 33 synchronous queues to serve. Becuase it does not know about groups, it has no choice but to serve them in round robin manner. So it does not sound like a CFQ issue. Just that CFQ can give random reader an advantage if it knows that random reader is in a different group and that's where IO controller comes in to picture.
Quoted 129 lines
Quoted 110 lines
- With IO controller, one can provide isolation to the random reader group and maintain consitent view of bandwidth and latencies.
Test2: Random Reader Vs Sequential Reader
========================================
Launched a random reader and then increasing number of sequential readers to see the effect on BW and latencies of random reader.[fio --rw=read --bs=4K --size=2G --runtime=30 --direct=1 --numjobs= <1 to 16> ] [fio --rw=randread --bs=4K --size=2G --runtime=30 --direct=1]
[ Vanilla CFQ, No groups ]
<---------------seq readers----------------------> <------random reader-->nr Max-bdwidth Min-bdwidth Agg-bdwidth Max-latency Agg-bdwidth Max-latency 1 23318KiB/s 23318KiB/s 23318KiB/s 55940 usec 36KiB/s 247K usec 2 14732KiB/s 11406KiB/s 26126KiB/s 142K usec 20KiB/s 446K usec 4 9417KiB/s 5169KiB/s 27338KiB/s 404K usec 10KiB/s 993K usec 8 3360KiB/s 3041KiB/s 25850KiB/s 954K usec 60KiB/s 956K usec 16 1888KiB/s 1457KiB/s 26763KiB/s 1871K usec 28KiB/s 1868K usecCreated two cgroups group1 and group2 of weights 500 each. Launched increasing number of sequential readers in group1 and one random reader in group2 using fio.
[IO controller CFQ; group_idle=1; group1 weight=500; group2 weight=500] <---------------group1---------------------------> <------group2--------->
nr Max-bdwidth Min-bdwidth Agg-bdwidth Max-latency Agg-bdwidth Max-latency 1 13733KiB/s 13733KiB/s 13733KiB/s 247K usec 330KiB/s 154K usec 2 8553KiB/s 4963KiB/s 13514KiB/s 472K usec 322KiB/s 174K usec 4 5045KiB/s 1367KiB/s 13134KiB/s 947K usec 318KiB/s 178K usec 8 1774KiB/s 1420KiB/s 13035KiB/s 1871K usec 323KiB/s 233K usec 16 959KiB/s 518KiB/s 12691KiB/s 3809K usec 324KiB/s 208K usecAlso ran the same test with IO controller CFQ in flat mode to see if there are any major deviations from Vanilla CFQ. Does not look like any.
[IO controller CFQ; No groups ]
<---------------seq readers----------------------> <------random reader-->nr Max-bdwidth Min-bdwidth Agg-bdwidth Max-latency Agg-bdwidth Max-latency 1 23028KiB/s 23028KiB/s 23028KiB/s 47460 usec 36KiB/s 253K usec 2 14452KiB/s 11176KiB/s 25628KiB/s 145K usec 20KiB/s 447K usec 4 8815KiB/s 5720KiB/s 27121KiB/s 396K usec 10KiB/s 968K usec 8 3335KiB/s 2827KiB/s 24866KiB/s 960K usec 62KiB/s 955K usec 16 1784KiB/s 1311KiB/s 26537KiB/s 1883K usec 26KiB/s 1866K usecNotes:
- The BW and latencies of random reader in group 2 seems to be stable and bounded and does not get impacted much as number of sequential readers increase in group1. Hence provding good isolation.- Throughput of sequential readers comes down and latencies go up as half of disk bandwidth (in terms of time) has been reserved for random reader group.
Test3: Sequential Reader Vs Sequential Reader
============================================
Created two cgroups group1 and group2 of weights 500 and 1000 respectively. Launched increasing number of sequential readers in group1 and one sequential reader in group2 using fio and monitored how bandwidth is being distributed between two groups.First 5 columns give stats about job in group1 and last two columns give stats about job in group2.
<---------------group1---------------------------> <------group2--------->
nr Max-bdwidth Min-bdwidth Agg-bdwidth Max-latency Agg-bdwidth Max-latency 1 8970KiB/s 8970KiB/s 8970KiB/s 230K usec 20681KiB/s 124K usec 2 6783KiB/s 3202KiB/s 9984KiB/s 546K usec 19682KiB/s 139K usec 4 4641KiB/s 1029KiB/s 9280KiB/s 1185K usec 19235KiB/s 172K usec 8 1435KiB/s 1079KiB/s 9926KiB/s 2461K usec 19501KiB/s 153K usec 16 764KiB/s 398KiB/s 9395KiB/s 4986K usec 19367KiB/s 172K usecNote: group2 is getting double the bandwidth of group1 even in the face of increasing number of readers in group1.
Test4 (Isolation between two KVM virtual machines)
==================================================
Created two KVM virtual machines. Partitioned a disk on host in two partitions and gave one partition to each virtual machine. Put both the virtual machines in two different cgroup of weight 1000 and 500 each. Virtual machines created ext3 file system on the partitions exported from host and did buffered writes. Host seems writes as synchronous and virtual machine with higher weight gets double the disk time of virtual machine of lower weight. Used deadline scheduler in this test case.Some more details about configuration are in documentation patch.
Test5 (Fairness for async writes, Buffered Write Vs Buffered Write)
===================================================================
Fairness for async writes is tricky and biggest reason is that async writes are cached in higher layers (page cahe) as well as possibly in file system layer also (btrfs, xfs etc), and are dispatched to lower layers not necessarily in proportional manner.For example, consider two dd threads reading
/dev/zeroas input file and doing writes of huge files. Very soon we will crossvm_dirty_ratioand dd thread will be forced to write out some pages to disk before more pages can be dirtied. But not necessarily dirty pages of same thread are picked. It can very well pick the inode of lesser priority dd thread and do some writeout. So effectively higher weight dd is doing writeouts of lower weight dd pages and we don't see service differentation.IOW, the core problem with buffered write fairness is that higher weight thread does not throw enought IO traffic at IO controller to keep the queue continuously backlogged. In my testing, there are many .2 to .8 second intervals where higher weight queue is empty and in that duration lower weight queue get lots of job done giving the impression that there was no service differentiation.
Quoted 5 lines
In summary, from IO controller point of view async writes support is there. Because page cache has not been designed in such a manner that higher prio/weight writer can do more write out as compared to lower prio/weight writer, gettting service differentiation is hard and it is visible in some cases and not visible in some cases.
Here's where it all falls to pieces.
For async writeback we just don't care about IO priorities. Because from the point of view of the userspace task, the write was async! It occurred at memory bandwidth speed.
It's only when the kernel's dirty memory thresholds start to get exceeded that we start to care about prioritisation. And at that time, all dirty memory (within a memcg?) is equal - a high-ioprio dirty page consumes just as much memory as a low-ioprio dirty page.
So when
balance_dirty_pages()hits, what do we want to do?
Quoted 7 lines
I suppose that all we can do is to block low-ioprio processes more agressively at the VFS layer, to reduce the rate at which they're dirtying memory so as to give high-ioprio processes more of the disk bandwidth.
But you've gone and implemented all of this stuff at the io-controller level and not at the VFS level so you're, umm, screwed.
True that's an issue. For async writes we don't create parallel IO paths from user space to IO scheduler hence it is hard to provide fairness in all the cases. I think part of the problem is page cache and some serialization also comes from kjournald.
How about coming up with another cgroup controller for buffered writes or clubbing it with memory controller as KAMEZAWA Hiroyuki suggested and co-mount this with io controller? This should help control buffered writes per cgroup.
Quoted 119 lines
Importantly screwed! It's a very common workload pattern, and one which causes tremendous amounts of IO to be generated very quickly, traditionally causing bad latency effects all over the place. And we have no answer to this.
Quoted 113 lines
Vanilla CFQ Vs IO Controller CFQ
================================
We have not fundamentally changed CFQ, instead enhanced it to also support hierarchical io scheduling. In the process invariably there are small changes here and there as new scenarios come up. Running some tests here and comparing both the CFQ's to see if there is any major deviation in behavior.Test1: Sequential Readers
=========================
[fio --rw=read --bs=4K --size=2G --runtime=30 --direct=1 --numjobs=<1 to 16> ]IO scheduler: Vanilla CFQ
nr Max-bdwidth Min-bdwidth Agg-bdwidth Max-latency 1 35499KiB/s 35499KiB/s 35499KiB/s 19195 usec 2 17089KiB/s 13600KiB/s 30690KiB/s 118K usec 4 9165KiB/s 5421KiB/s 29411KiB/s 380K usec 8 3815KiB/s 3423KiB/s 29312KiB/s 830K usec 16 1911KiB/s 1554KiB/s 28921KiB/s 1756K usecIO scheduler: IO controller CFQ
nr Max-bdwidth Min-bdwidth Agg-bdwidth Max-latency 1 34494KiB/s 34494KiB/s 34494KiB/s 14482 usec 2 16983KiB/s 13632KiB/s 30616KiB/s 123K usec 4 9237KiB/s 5809KiB/s 29631KiB/s 372K usec 8 3901KiB/s 3505KiB/s 29162KiB/s 822K usec 16 1895KiB/s 1653KiB/s 28945KiB/s 1778K usecTest2: Sequential Writers
=========================
[fio --rw=write --bs=4K --size=2G --runtime=30 --direct=1 --numjobs=<1 to 16> ]IO scheduler: Vanilla CFQ
nr Max-bdwidth Min-bdwidth Agg-bdwidth Max-latency 1 22669KiB/s 22669KiB/s 22669KiB/s 401K usec 2 14760KiB/s 7419KiB/s 22179KiB/s 571K usec 4 5862KiB/s 5746KiB/s 23174KiB/s 444K usec 8 3377KiB/s 2199KiB/s 22427KiB/s 1057K usec 16 2229KiB/s 556KiB/s 20601KiB/s 5099K usecIO scheduler: IO Controller CFQ
nr Max-bdwidth Min-bdwidth Agg-bdwidth Max-latency 1 22911KiB/s 22911KiB/s 22911KiB/s 37319 usec 2 11752KiB/s 11632KiB/s 23383KiB/s 245K usec 4 6663KiB/s 5409KiB/s 23207KiB/s 384K usec 8 3161KiB/s 2460KiB/s 22566KiB/s 935K usec 16 1888KiB/s 795KiB/s 21349KiB/s 3009K usecTest3: Random Readers
=========================
[fio --rw=randread --bs=4K --size=2G --runtime=30 --direct=1 --numjobs=1 to 16]IO scheduler: Vanilla CFQ
nr Max-bdwidth Min-bdwidth Agg-bdwidth Max-latency 1 484KiB/s 484KiB/s 484KiB/s 22596 usec 2 229KiB/s 196KiB/s 425KiB/s 51111 usec 4 119KiB/s 73KiB/s 405KiB/s 2344 msec 8 93KiB/s 23KiB/s 399KiB/s 2246 msec 16 38KiB/s 8KiB/s 328KiB/s 3965 msecIO scheduler: IO Controller CFQ
nr Max-bdwidth Min-bdwidth Agg-bdwidth Max-latency 1 483KiB/s 483KiB/s 483KiB/s 29391 usec 2 229KiB/s 196KiB/s 426KiB/s 51625 usec 4 132KiB/s 88KiB/s 417KiB/s 2313 msec 8 79KiB/s 18KiB/s 389KiB/s 2298 msec 16 43KiB/s 9KiB/s 327KiB/s 3905 msecTest4: Random Writers
=====================
[fio --rw=randwrite --bs=4K --size=2G --runtime=30 --direct=1 --numjobs=1 to 16]IO scheduler: Vanilla CFQ
nr Max-bdwidth Min-bdwidth Agg-bdwidth Max-latency 1 14641KiB/s 14641KiB/s 14641KiB/s 93045 usec 2 7896KiB/s 1348KiB/s 9245KiB/s 82778 usec 4 2657KiB/s 265KiB/s 6025KiB/s 216K usec 8 951KiB/s 122KiB/s 3386KiB/s 1148K usec 16 66KiB/s 22KiB/s 829KiB/s 1308 msecIO scheduler: IO Controller CFQ
nr Max-bdwidth Min-bdwidth Agg-bdwidth Max-latency 1 14454KiB/s 14454KiB/s 14454KiB/s 74623 usec 2 4595KiB/s 4104KiB/s 8699KiB/s 135K usec 4 3113KiB/s 334KiB/s 5782KiB/s 200K usec 8 1146KiB/s 95KiB/s 3832KiB/s 593K usec 16 71KiB/s 29KiB/s 814KiB/s 1457 msecNotes:
- Does not look like that anything has changed significantly.Previous versions of the patches were posted here.
------------------------------------------------(V1) http://lkml.org/lkml/2009/3/11/486
(V2) http://lkml.org/lkml/2009/5/5/275
(V3) http://lkml.org/lkml/2009/5/26/472
(V4) http://lkml.org/lkml/2009/6/8/580
(V5) http://lkml.org/lkml/2009/6/19/279
(V6) http://lkml.org/lkml/2009/7/2/369
(V7) http://lkml.org/lkml/2009/7/24/253
(V8) http://lkml.org/lkml/2009/8/16/204
(V9) http://lkml.org/lkml/2009/8/28/327Thanks
Vivek
Hi Vivek,
Quoted 11 lines · Vivek Goyal
Higher level solutions are not keeping track of time slices. Time slices will be allocated by CFQ which does not have any idea about grouping. Higher level controller just keeps track of size of IO done at group level and then run either a leaky bucket or token bucket algorithm.
IO throttling is a max BW controller, so it will not even care about what is happening in other group. It will just be concerned with rate of IO in one particular group and if we exceed specified limit, throttle it. So until and unless sequential reader group hits it max bw limit, it will keep sending reads down to CFQ, and CFQ will happily assign 100ms slices to readers.
Quoted 4 lines
dm-ioband will not try to choke the high throughput sequential reader group for the slow random reader group because that would just kill the throughput of rotational media. Every sequential reader will run for few ms and then be throttled and this goes on. Disk will soon be seek bound.
Because dm-ioband provides faireness in terms of how many IO requests are issued or how many bytes are transferred, so this behaviour is to be expected. Do you think fairness in terms of IO requests and size is not fair?
Quoted 30 lines
Quoted 8 lines
Quoted 5 lines
Buffering at higher layer can delay read requests for more than slice idle period of CFQ (default 8 ms). That means, it is possible that we are waiting for a request from the queue but it is buffered at higher layer and then idle timer will fire. It means that queue will losse its share at the same time overall throughput will be impacted as we lost those 8 ms.
That sounds like a bug.
Actually this probably is a limitation of higher level controller. It most likely is sitting so high in IO stack that it has no idea what underlying IO scheduler is and what are IO scheduler's policies. So it can't keep up with IO scheduler's policies. Secondly, it might be a low weight group and tokens might not be available fast enough to release the request.
Quoted 11 lines
Quoted 10 lines
Read Vs Write
-------------
Writes can overwhelm readers hence second level controller FIFO release will run into issue here. If there is a single queue maintained then reads will suffer large latencies. If there separate queues for reads and writes then it will be hard to decide in what ratio to dispatch reads and writes as it is IO scheduler's decision to decide when and how much read/write to dispatch. This is another place where higher level controller will not be in sync with lower level io scheduler and can change the effective policies of underlying io scheduler.Quoted 3 lines
The IO schedulers already take care of read-vs-write and already take care of preventing large writes-starve-reads latencies (or at least, they're supposed to).
Quoted 7 lines
True. Actually this is a limitation of higher level controller. A higher level controller will most likely implement some of kind of queuing/buffering mechanism where it will buffer requeuests when it decides to throttle the group. Now once a fair number read and requests are buffered, and if controller is ready to dispatch some requests from the group, which requests/bio should it dispatch? reads first or writes first or reads and writes in certain ratio?
The write-starve-reads on dm-ioband, that you pointed out before, was not caused by FIFO release, it was caused by IO flow control in dm-ioband. When I turned off the flow control, then the read throughput was quite improved.
Now I'm considering separating dm-ioband's internal queue into sync and async and giving a certain priority of dispatch to async IOs.
Thanks,
Ryo Tsuruta
Quoted 19 lines · Ryo Tsuruta
Hi Vivek,
Quoted 11 lines · Vivek Goyal
Higher level solutions are not keeping track of time slices. Time slices will be allocated by CFQ which does not have any idea about grouping. Higher level controller just keeps track of size of IO done at group level and then run either a leaky bucket or token bucket algorithm.
IO throttling is a max BW controller, so it will not even care about what is happening in other group. It will just be concerned with rate of IO in one particular group and if we exceed specified limit, throttle it. So until and unless sequential reader group hits it max bw limit, it will keep sending reads down to CFQ, and CFQ will happily assign 100ms slices to readers.
Quoted 4 lines
dm-ioband will not try to choke the high throughput sequential reader group for the slow random reader group because that would just kill the throughput of rotational media. Every sequential reader will run for few ms and then be throttled and this goes on. Disk will soon be seek bound.
Quoted 5 lines
Because dm-ioband provides faireness in terms of how many IO requests are issued or how many bytes are transferred, so this behaviour is to be expected. Do you think fairness in terms of IO requests and size is not fair?
Hi Ryo,
Fairness in terms of size of IO or number of requests is probably not the best thing to do on rotational media where seek latencies are significant.
It probably should work just well on media with very low seek latencies like SSD.
So on rotational media, either you will not provide fairness to random readers because they are too slow or you will choke the sequential readers in other group and also bring down the overall disk throughput.
If you don't decide to choke/throttle sequential reader group for the sake of random reader in other group then you will not have a good control on random reader latencies. Because now IO scheduler sees the IO from both sequential reader as well as random reader and sequential readers have not been throttled. So the dispatch pattern/time slices will again look like..
SR1 SR2 SR3 SR4 SR5 RR..... instead of SR1 RR SR2 RR SR3 RR SR4 RR ....
SR --> sequential reader, RR --> random reader
Quoted 38 lines
Quoted 30 lines
Quoted 8 lines
Quoted 5 lines
Buffering at higher layer can delay read requests for more than slice idle period of CFQ (default 8 ms). That means, it is possible that we are waiting for a request from the queue but it is buffered at higher layer and then idle timer will fire. It means that queue will losse its share at the same time overall throughput will be impacted as we lost those 8 ms.
That sounds like a bug.
Actually this probably is a limitation of higher level controller. It most likely is sitting so high in IO stack that it has no idea what underlying IO scheduler is and what are IO scheduler's policies. So it can't keep up with IO scheduler's policies. Secondly, it might be a low weight group and tokens might not be available fast enough to release the request.
Quoted 11 lines
Quoted 10 lines
Read Vs Write
-------------
Writes can overwhelm readers hence second level controller FIFO release will run into issue here. If there is a single queue maintained then reads will suffer large latencies. If there separate queues for reads and writes then it will be hard to decide in what ratio to dispatch reads and writes as it is IO scheduler's decision to decide when and how much read/write to dispatch. This is another place where higher level controller will not be in sync with lower level io scheduler and can change the effective policies of underlying io scheduler.Quoted 3 lines
The IO schedulers already take care of read-vs-write and already take care of preventing large writes-starve-reads latencies (or at least, they're supposed to).
Quoted 7 lines
True. Actually this is a limitation of higher level controller. A higher level controller will most likely implement some of kind of queuing/buffering mechanism where it will buffer requeuests when it decides to throttle the group. Now once a fair number read and requests are buffered, and if controller is ready to dispatch some requests from the group, which requests/bio should it dispatch? reads first or writes first or reads and writes in certain ratio?
Quoted 4 lines
The write-starve-reads on dm-ioband, that you pointed out before, was not caused by FIFO release, it was caused by IO flow control in dm-ioband. When I turned off the flow control, then the read throughput was quite improved.
What was flow control doing?
Quoted 3 lines
Now I'm considering separating dm-ioband's internal queue into sync and async and giving a certain priority of dispatch to async IOs.
Even if you maintain separate queues for sync and async, in what ratio will you dispatch reads and writes to underlying layer once fresh tokens become available to the group and you decide to unthrottle the group.
Whatever policy you adopt for read and write dispatch, it might not match with policy of underlying IO scheduler because every IO scheduler seems to have its own way of determining how reads and writes should be dispatched.
Now somebody might start complaining that my job inside the group is not getting same reader/writer ratio as it was getting outside the group.
Thanks
Vivek
Hi Vivek,
Quoted 24 lines · Vivek Goyal
Quoted 5 lines
Because dm-ioband provides faireness in terms of how many IO requests are issued or how many bytes are transferred, so this behaviour is to be expected. Do you think fairness in terms of IO requests and size is not fair?
Hi Ryo,
Fairness in terms of size of IO or number of requests is probably not the best thing to do on rotational media where seek latencies are significant.
It probably should work just well on media with very low seek latencies like SSD.
So on rotational media, either you will not provide fairness to random readers because they are too slow or you will choke the sequential readers in other group and also bring down the overall disk throughput.
If you don't decide to choke/throttle sequential reader group for the sake of random reader in other group then you will not have a good control on random reader latencies. Because now IO scheduler sees the IO from both sequential reader as well as random reader and sequential readers have not been throttled. So the dispatch pattern/time slices will again look like..
Quoted 7 lines
SR1 SR2 SR3 SR4 SR5 RR..... instead of SR1 RR SR2 RR SR3 RR SR4 RR ....SR --> sequential reader, RR --> random reader
Thank you for elaborating. However, I think that fairness in terms of disk time has a similar problem. The below is a benchmark result of randread vs seqread I posted before, rand-readers and seq-readers ran on individual groups and their weights were equally assigned.
Throughput [KiB/s]
io-controller dm-ioband
randread 161 314
seqread 9556 631
I know that dm-ioband is needed to improvement on the seqread throughput, but I don't think that io-controller seems quite fair, even the disk times of each group are equal, why randread can't get more bandwidth. So I think that this is how users think about faireness, and it would be good thing to provide multiple policies of bandwidth control for uses.
Quoted 6 lines
Quoted 4 lines
The write-starve-reads on dm-ioband, that you pointed out before, was not caused by FIFO release, it was caused by IO flow control in dm-ioband. When I turned off the flow control, then the read throughput was quite improved.
What was flow control doing?
dm-ioband gives a limit on each IO group. When the number of IO requests backlogged in a group exceeds the limit, processes which are going to issue IO requests to the group are made sleep until all the backlogged requests are flushed out.
Quoted 6 lines
Quoted 2 lines
Now I'm considering separating dm-ioband's internal queue into sync and async and giving a certain priority of dispatch to async IOs.
Even if you maintain separate queues for sync and async, in what ratio will you dispatch reads and writes to underlying layer once fresh tokens become available to the group and you decide to unthrottle the group.
Now I'm thinking that It's according to the requested order, but when the number of in-flight sync IOs exceeds io_limit (io_limit is calculated based on nr_requests of underlying block device), dm-ioband dispatches only async IOs until the number of in-flight sync IOs are below the io_limit, and vice versa. At least it could solve the write-starve-read issue which you pointed out.
Quoted 3 lines
Whatever policy you adopt for read and write dispatch, it might not match with policy of underlying IO scheduler because every IO scheduler seems to have its own way of determining how reads and writes should be dispatched.
I think that this is a matter of users choise, which a user would like to give priority to bandwidth or IO scheduler's policy.
Quoted 5 lines
Now somebody might start complaining that my job inside the group is not getting same reader/writer ratio as it was getting outside the group.
Thanks
Vivek
Thanks,
Ryo Tsuruta
A message that is not in the archive (sent off-list or not yet received): [email protected]
Hi Vivek,
Quoted 24 lines · Vivek Goyal
Quoted 5 lines
Because dm-ioband provides faireness in terms of how many IO requests are issued or how many bytes are transferred, so this behaviour is to be expected. Do you think fairness in terms of IO requests and size is not fair?
Hi Ryo,
Fairness in terms of size of IO or number of requests is probably not the best thing to do on rotational media where seek latencies are significant.
It probably should work just well on media with very low seek latencies like SSD.
So on rotational media, either you will not provide fairness to random readers because they are too slow or you will choke the sequential readers in other group and also bring down the overall disk throughput.
If you don't decide to choke/throttle sequential reader group for the sake of random reader in other group then you will not have a good control on random reader latencies. Because now IO scheduler sees the IO from both sequential reader as well as random reader and sequential readers have not been throttled. So the dispatch pattern/time slices will again look like..
Quoted 7 lines
SR1 SR2 SR3 SR4 SR5 RR..... instead of SR1 RR SR2 RR SR3 RR SR4 RR ....SR --> sequential reader, RR --> random reader
Thank you for elaborating. However, I think that fairness in terms of disk time has a similar problem. The below is a benchmark result of randread vs seqread I posted before, rand-readers and seq-readers ran on individual groups and their weights were equally assigned.
Throughput [KiB/s]
io-controller dm-ioband
randread 161 314
seqread 9556 631
I know that dm-ioband is needed to improvement on the seqread throughput, but I don't think that io-controller seems quite fair, even the disk times of each group are equal, why randread can't get more bandwidth. So I think that this is how users think about faireness, and it would be good thing to provide multiple policies of bandwidth control for uses.
Quoted 6 lines
Quoted 4 lines
The write-starve-reads on dm-ioband, that you pointed out before, was not caused by FIFO release, it was caused by IO flow control in dm-ioband. When I turned off the flow control, then the read throughput was quite improved.
What was flow control doing?
dm-ioband gives a limit on each IO group. When the number of IO requests backlogged in a group exceeds the limit, processes which are going to issue IO requests to the group are made sleep until all the backlogged requests are flushed out.
Quoted 6 lines
Quoted 2 lines
Now I'm considering separating dm-ioband's internal queue into sync and async and giving a certain priority of dispatch to async IOs.
Even if you maintain separate queues for sync and async, in what ratio will you dispatch reads and writes to underlying layer once fresh tokens become available to the group and you decide to unthrottle the group.
Now I'm thinking that It's according to the requested order, but when the number of in-flight sync IOs exceeds io_limit (io_limit is calculated based on nr_requests of underlying block device), dm-ioband dispatches only async IOs until the number of in-flight sync IOs are below the io_limit, and vice versa. At least it could solve the write-starve-read issue which you pointed out.
Quoted 3 lines
Whatever policy you adopt for read and write dispatch, it might not match with policy of underlying IO scheduler because every IO scheduler seems to have its own way of determining how reads and writes should be dispatched.
I think that this is a matter of users choise, which a user would like to give priority to bandwidth or IO scheduler's policy.
Quoted 5 lines
Now somebody might start complaining that my job inside the group is not getting same reader/writer ratio as it was getting outside the group.
Thanks
Vivek
Thanks,
Ryo Tsuruta
A message that is not in the archive (sent off-list or not yet received): 20090925.180724.104041942.ryov-jCdQPDEk3idL9jVzuh4AOg@public.gmane.org
Quoted 19 lines · Ryo Tsuruta
Hi Vivek,
Quoted 11 lines · Vivek Goyal
Higher level solutions are not keeping track of time slices. Time slices will be allocated by CFQ which does not have any idea about grouping. Higher level controller just keeps track of size of IO done at group level and then run either a leaky bucket or token bucket algorithm.
IO throttling is a max BW controller, so it will not even care about what is happening in other group. It will just be concerned with rate of IO in one particular group and if we exceed specified limit, throttle it. So until and unless sequential reader group hits it max bw limit, it will keep sending reads down to CFQ, and CFQ will happily assign 100ms slices to readers.
Quoted 4 lines
dm-ioband will not try to choke the high throughput sequential reader group for the slow random reader group because that would just kill the throughput of rotational media. Every sequential reader will run for few ms and then be throttled and this goes on. Disk will soon be seek bound.
Quoted 5 lines
Because dm-ioband provides faireness in terms of how many IO requests are issued or how many bytes are transferred, so this behaviour is to be expected. Do you think fairness in terms of IO requests and size is not fair?
Hi Ryo,
Fairness in terms of size of IO or number of requests is probably not the best thing to do on rotational media where seek latencies are significant.
It probably should work just well on media with very low seek latencies like SSD.
So on rotational media, either you will not provide fairness to random readers because they are too slow or you will choke the sequential readers in other group and also bring down the overall disk throughput.
If you don't decide to choke/throttle sequential reader group for the sake of random reader in other group then you will not have a good control on random reader latencies. Because now IO scheduler sees the IO from both sequential reader as well as random reader and sequential readers have not been throttled. So the dispatch pattern/time slices will again look like..
SR1 SR2 SR3 SR4 SR5 RR..... instead of SR1 RR SR2 RR SR3 RR SR4 RR ....
SR --> sequential reader, RR --> random reader
Quoted 38 lines
Quoted 30 lines
Quoted 8 lines
Quoted 5 lines
Buffering at higher layer can delay read requests for more than slice idle period of CFQ (default 8 ms). That means, it is possible that we are waiting for a request from the queue but it is buffered at higher layer and then idle timer will fire. It means that queue will losse its share at the same time overall throughput will be impacted as we lost those 8 ms.
That sounds like a bug.
Actually this probably is a limitation of higher level controller. It most likely is sitting so high in IO stack that it has no idea what underlying IO scheduler is and what are IO scheduler's policies. So it can't keep up with IO scheduler's policies. Secondly, it might be a low weight group and tokens might not be available fast enough to release the request.
Quoted 11 lines
Quoted 10 lines
Read Vs Write
-------------
Writes can overwhelm readers hence second level controller FIFO release will run into issue here. If there is a single queue maintained then reads will suffer large latencies. If there separate queues for reads and writes then it will be hard to decide in what ratio to dispatch reads and writes as it is IO scheduler's decision to decide when and how much read/write to dispatch. This is another place where higher level controller will not be in sync with lower level io scheduler and can change the effective policies of underlying io scheduler.Quoted 3 lines
The IO schedulers already take care of read-vs-write and already take care of preventing large writes-starve-reads latencies (or at least, they're supposed to).
Quoted 7 lines
True. Actually this is a limitation of higher level controller. A higher level controller will most likely implement some of kind of queuing/buffering mechanism where it will buffer requeuests when it decides to throttle the group. Now once a fair number read and requests are buffered, and if controller is ready to dispatch some requests from the group, which requests/bio should it dispatch? reads first or writes first or reads and writes in certain ratio?
Quoted 4 lines
The write-starve-reads on dm-ioband, that you pointed out before, was not caused by FIFO release, it was caused by IO flow control in dm-ioband. When I turned off the flow control, then the read throughput was quite improved.
What was flow control doing?
Quoted 3 lines
Now I'm considering separating dm-ioband's internal queue into sync and async and giving a certain priority of dispatch to async IOs.
Even if you maintain separate queues for sync and async, in what ratio will you dispatch reads and writes to underlying layer once fresh tokens become available to the group and you decide to unthrottle the group.
Whatever policy you adopt for read and write dispatch, it might not match with policy of underlying IO scheduler because every IO scheduler seems to have its own way of determining how reads and writes should be dispatched.
Now somebody might start complaining that my job inside the group is not getting same reader/writer ratio as it was getting outside the group.
Thanks
Vivek
Quoted 4 lines · Ryo Tsuruta
Because dm-ioband provides faireness in terms of how many IO requests are issued or how many bytes are transferred, so this behaviour is to be expected. Do you think fairness in terms of IO requests and size is not fair?
When there are two workloads competing for the same resources, I would expect each of the workloads to run at about 50% of the speed at which it would run on an uncontended system.
Having one of the workloads run at 95% of the uncontended speed and the other workload at 5% is "not fair" (to put it diplomatically).
Signature · 1 line
All rights reversed.
Quoted 4 lines · Ryo Tsuruta
Because dm-ioband provides faireness in terms of how many IO requests are issued or how many bytes are transferred, so this behaviour is to be expected. Do you think fairness in terms of IO requests and size is not fair?
When there are two workloads competing for the same resources, I would expect each of the workloads to run at about 50% of the speed at which it would run on an uncontended system.
Having one of the workloads run at 95% of the uncontended speed and the other workload at 5% is "not fair" (to put it diplomatically).
Signature · 1 line
All rights reversed.
Hi Rik,
Quoted 7 lines · Rik van Riel
Quoted 4 lines · Ryo Tsuruta
Because dm-ioband provides faireness in terms of how many IO requests are issued or how many bytes are transferred, so this behaviour is to be expected. Do you think fairness in terms of IO requests and size is not fair?
Quoted 8 lines
When there are two workloads competing for the same resources, I would expect each of the workloads to run at about 50% of the speed at which it would run on an uncontended system.
Having one of the workloads run at 95% of the uncontended speed and the other workload at 5% is "not fair" (to put it diplomatically).
As I wrote in the mail to Vivek, I think that providing multiple policies, on a per disk time basis, on a per iosize basis, maximum rate limiting or etc would be good for users.
Thanks,
Ryo Tsuruta
A message that is not in the archive (sent off-list or not yet received): [email protected]
Hi Rik,
Quoted 7 lines · Rik van Riel
Quoted 4 lines · Ryo Tsuruta
Because dm-ioband provides faireness in terms of how many IO requests are issued or how many bytes are transferred, so this behaviour is to be expected. Do you think fairness in terms of IO requests and size is not fair?
Quoted 8 lines
When there are two workloads competing for the same resources, I would expect each of the workloads to run at about 50% of the speed at which it would run on an uncontended system.
Having one of the workloads run at 95% of the uncontended speed and the other workload at 5% is "not fair" (to put it diplomatically).
As I wrote in the mail to Vivek, I think that providing multiple policies, on a per disk time basis, on a per iosize basis, maximum rate limiting or etc would be good for users.
Thanks,
Ryo Tsuruta
A message that is not in the archive (sent off-list or not yet received): [email protected]
Hi Vivek,
Quoted 11 lines · Vivek Goyal
Higher level solutions are not keeping track of time slices. Time slices will be allocated by CFQ which does not have any idea about grouping. Higher level controller just keeps track of size of IO done at group level and then run either a leaky bucket or token bucket algorithm.
IO throttling is a max BW controller, so it will not even care about what is happening in other group. It will just be concerned with rate of IO in one particular group and if we exceed specified limit, throttle it. So until and unless sequential reader group hits it max bw limit, it will keep sending reads down to CFQ, and CFQ will happily assign 100ms slices to readers.
Quoted 4 lines
dm-ioband will not try to choke the high throughput sequential reader group for the slow random reader group because that would just kill the throughput of rotational media. Every sequential reader will run for few ms and then be throttled and this goes on. Disk will soon be seek bound.
Because dm-ioband provides faireness in terms of how many IO requests are issued or how many bytes are transferred, so this behaviour is to be expected. Do you think fairness in terms of IO requests and size is not fair?
Quoted 30 lines
Quoted 8 lines
Quoted 5 lines
Buffering at higher layer can delay read requests for more than slice idle period of CFQ (default 8 ms). That means, it is possible that we are waiting for a request from the queue but it is buffered at higher layer and then idle timer will fire. It means that queue will losse its share at the same time overall throughput will be impacted as we lost those 8 ms.
That sounds like a bug.
Actually this probably is a limitation of higher level controller. It most likely is sitting so high in IO stack that it has no idea what underlying IO scheduler is and what are IO scheduler's policies. So it can't keep up with IO scheduler's policies. Secondly, it might be a low weight group and tokens might not be available fast enough to release the request.
Quoted 11 lines
Quoted 10 lines
Read Vs Write
-------------
Writes can overwhelm readers hence second level controller FIFO release will run into issue here. If there is a single queue maintained then reads will suffer large latencies. If there separate queues for reads and writes then it will be hard to decide in what ratio to dispatch reads and writes as it is IO scheduler's decision to decide when and how much read/write to dispatch. This is another place where higher level controller will not be in sync with lower level io scheduler and can change the effective policies of underlying io scheduler.Quoted 3 lines
The IO schedulers already take care of read-vs-write and already take care of preventing large writes-starve-reads latencies (or at least, they're supposed to).
Quoted 7 lines
True. Actually this is a limitation of higher level controller. A higher level controller will most likely implement some of kind of queuing/buffering mechanism where it will buffer requeuests when it decides to throttle the group. Now once a fair number read and requests are buffered, and if controller is ready to dispatch some requests from the group, which requests/bio should it dispatch? reads first or writes first or reads and writes in certain ratio?
The write-starve-reads on dm-ioband, that you pointed out before, was not caused by FIFO release, it was caused by IO flow control in dm-ioband. When I turned off the flow control, then the read throughput was quite improved.
Now I'm considering separating dm-ioband's internal queue into sync and async and giving a certain priority of dispatch to async IOs.
Thanks,
Ryo Tsuruta
More performance numbers (Was: Re: IO scheduler based IO controller V10)
In reply to Andrew MortonOn Thu, Sep 24, 2009 at 02:33:15PM -0700, Andrew Morton wrote:
[..]
Quoted 4 lines
Quoted 4 lines
Testing
=======
Quoted 10 lines
Quoted 3 lines
Environment
==========
A 7200 RPM SATA drive with queue depth of 31. Ext3 filesystem.That's a bit of a toy.
Do we have testing results for more enterprisey hardware? Big storage arrays? SSD? Infiniband? iscsi? nfs? (lol, gotcha)
Hi Andrew,
I got hold of a relatively more enterprisey stuff. It is an storage array with few striped disks(I think 4 or 5). So this is not high end stuff but better than my single SATA disk. I guess may be entry level enterprisy stuff. Still trying to get hold of higher end configuration...
Apart from IO scheduler controller number, I also got a chance to run same tests with dm-ioband controller. I am posting these too. I am also planning to run similar numbers on Andrea's "max bw" controller also. Should be able to post those numbers also in 2-3 days.
Software Environment
====================
- 2.6.31 kernel
- V10 of IO scheduler based controller
- version v1.14.0 of dm-ioband patches
Used fio jobs for 30 seconds in various configurations. All the IO is direct IO to eliminate the effects of caches.
I have run three sets for each test. Blindly reporting results of set2 from each test, otherwise it is too much of data to report.
Had lun of 2500GB capacity. Used 200G partitions with ext3 file system for my testing. For IO scheduler based controller patches, created two cgroups of weight 100 each doing IO to single 200G partition.
For dm-ioband, created two partitions of 200G each and created two ioband devices of weight 100 each with policy "weight-iosize". Ideally I should haved used cgroups on dm-ioband also but could not get cgroup patch going. Because this is striped configuration, not expecting any major changes in results due to that.
Sequential reader vs Random reader
==================================
Launched on random reader in one group and launched increasing number of sequential readers in other group to see the effect on latency and bandwidth of random reader.
[fio1 --rw=read --bs=4K --size=2G --runtime=30 --direct=1 ]
[fio2 --rw=randread --bs=4K --size=1G --runtime=30 --direct=1 --group_reporting]
Vanilla CFQ
-----------
[Sequential readers] [Random Reader]
nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 13806KB/s 13806KB/s 13483KB/s 28672 usec 1 23KB/s 212 msec 2 6406KB/s 6268KB/s 12378KB/s 128K usec 1 10KB/s 453 msec 4 3934KB/s 2536KB/s 13103KB/s 321K usec 1 6KB/s 847 msec 8 1934KB/s 556KB/s 13009KB/s 876K usec 1 13KB/s 1632 msec 16 958KB/s 280KB/s 13761KB/s 1621K usec 1 10KB/s 3217 msec 32 512KB/s 126KB/s 13861KB/s 3241K usec 1 6KB/s 3249 msec
IO scheduler controller + CFQ
-----------------------------
[Sequential readers] [Random Reader]
nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 5651KB/s 5651KB/s 5519KB/s 126K usec 1 222KB/s 130K usec 2 3144KB/s 1479KB/s 4515KB/s 347K usec 1 225KB/s 189K usec 4 1852KB/s 626KB/s 5128KB/s 775K usec 1 224KB/s 159K usec 8 971KB/s 279KB/s 6464KB/s 1666K usec 1 222KB/s 193K usec 16 454KB/s 129KB/s 6293KB/s 3356K usec 1 218KB/s 466K usec 32 239KB/s 42KB/s 5986KB/s 6753K usec 1 214KB/s 503K usec
Notes:
- The BW and latency of random reader are fairly stable in the face of increasing number of sequential readers. There are couple of spikes in latency, i guess comes from the hardware somehow. But will debug more to make sure that I am not delaying in dispatch of request.
dm-ioaband + CFQ
----------------
[Sequential readers] [Random Reader]
nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 12466KB/s 12466KB/s 12174KB/s 40078 usec 1 37KB/s 221 msec 2 6240KB/s 5904KB/s 11859KB/s 134K usec 1 12KB/s 443 msec 4 3517KB/s 2529KB/s 12368KB/s 357K usec 1 6KB/s 772 msec 8 1779KB/s 594KB/s 9857KB/s 719K usec 1 60KB/s 852K usec 16 914KB/s 300KB/s 10934KB/s 1467K usec 1 40KB/s 1285K usec 32 589KB/s 187KB/s 11537KB/s 3547K usec 1 14KB/s 3228 msec
Notes:
- Does not look like we provide fairness to random reader here. Latencies are on the rise and BW is on the decline. this is almost like Vanilla CFQ with reduced overall throughput.
- dm-ioband claims that they do not provide fairness for slow moving group and I think it is a bad idea. This leads to very weak isolation with no benefits. Especially if a buffered writer is running in other group. This should be fixed.
Random writers vs Random reader
================================
[fio1 --rw=randwrite --bs=64K --size=2G --runtime=30 --ioengine=libaio --iodepth=4 --direct=1 ] [fio2 --rw=randread --bs=4K --size=1G --runtime=30 --direct=1 --group_reporting]
Vanilla CFQ
-----------
[Random Writers] [Random Reader]
nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 67785KB/s 67785KB/s 66197KB/s 45499 usec 1 170KB/s 94098 usec 2 35163KB/s 35163KB/s 68678KB/s 218K usec 1 75KB/s 2335 msec 4 17759KB/s 15308KB/s 64206KB/s 2387K usec 1 85KB/s 2331 msec 8 8725KB/s 6495KB/s 57120KB/s 3761K usec 1 67KB/s 2488K usec 16 3912KB/s 3456KB/s 57121KB/s 1273K usec 1 60KB/s 1668K usec 32 2020KB/s 1503KB/s 56786KB/s 4221K usec 1 39KB/s 1101 msec
IO scheduler controller + CFQ
-----------------------------
[Random Writers] [Random Reader]
nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 20919KB/s 20919KB/s 20428KB/s 288K usec 1 213KB/s 580K usec 2 14765KB/s 14674KB/s 28749KB/s 776K usec 1 203KB/s 112K usec 4 7177KB/s 7091KB/s 27839KB/s 970K usec 1 197KB/s 132K usec 8 3027KB/s 2953KB/s 23285KB/s 3145K usec 1 218KB/s 203K usec 16 1959KB/s 1750KB/s 28919KB/s 1266K usec 1 160KB/s 182K usec 32 908KB/s 753KB/s 26267KB/s 2091K usec 1 208KB/s 144K usec
Notes:
- Again disk time has been divided half and half between random reader group and random writer group. Fairly stable BW and latencies for random reader in the face of increasing number of random writers.
- Drop in aggregate bw of random writers is expected as they now get only half of disk time.
dm-ioaband + CFQ
----------------
[Random Writers] [Random Reader]
nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 63659KB/s 63659KB/s 62167KB/s 89954 usec 1 164KB/s 72 msec 2 27109KB/s 27096KB/s 52933KB/s 674K usec 1 140KB/s 2204K usec 4 16553KB/s 16216KB/s 63946KB/s 694K usec 1 56KB/s 1871 msec 8 3907KB/s 3347KB/s 28752KB/s 2406K usec 1 226KB/s 2407K usec 16 2841KB/s 2647KB/s 42334KB/s 870K usec 1 52KB/s 3043 msec 32 738KB/s 657KB/s 21285KB/s 1529K usec 1 21KB/s 4435 msec
Notes:
- Again no fairness for random reader. Decreasing BW, increasing latency. No isolation in this case.
- I am curious what happened to random writer throughput in case of "32" writers. We did not get higher BW for random reader but random writer still suffering in throughput for random writer. I can see this for all the three sets.
Sequential Readers vs Sequential reader
=======================================
[fio1 --rw=read --bs=4K --size=2G --runtime=30 --direct=1] [fio2 --rw=read --bs=4K --size=2G --runtime=30 --direct=1]
Vanilla CFQ
-----------
[Sequential Readers] [Sequential Reader]
nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 6434KB/s 6434KB/s 6283KB/s 107K usec 1 7017KB/s 111K usec 2 4688KB/s 3284KB/s 7785KB/s 274K usec 1 4541KB/s 218K usec 4 3365KB/s 1326KB/s 9769KB/s 597K usec 1 3038KB/s 424K usec 8 1827KB/s 504KB/s 12053KB/s 813K usec 1 1389KB/s 813K usec 16 1022KB/s 301KB/s 13954KB/s 1618K usec 1 676KB/s 1617K usec 32 494KB/s 149KB/s 13611KB/s 3216K usec 1 416KB/s 3215K usec
IO scheduler controller + CFQ
-----------------------------
[Sequential Readers] [Sequential Reader]
nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 6605KB/s 6605KB/s 6450KB/s 120K usec 1 6527KB/s 120K usec 2 3706KB/s 1985KB/s 5558KB/s 323K usec 1 6331KB/s 149K usec 4 2053KB/s 672KB/s 5731KB/s 721K usec 1 6267KB/s 148K usec 8 1013KB/s 337KB/s 6962KB/s 1525K usec 1 6136KB/s 120K usec 16 497KB/s 125KB/s 6873KB/s 3226K usec 1 5882KB/s 113K usec 32 297KB/s 48KB/s 6445KB/s 6394K usec 1 5767KB/s 116K usec
Notes:
- Stable BW and lateneis for sequential reader in the face of increasing number of readers in other group.
dm-ioaband + CFQ
----------------
[Sequential Readers] [Sequential Reader]
nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 7140KB/s 7140KB/s 6972KB/s 112K usec 1 6886KB/s 165K usec 2 3965KB/s 2762KB/s 6569KB/s 479K usec 1 5887KB/s 475K usec 4 2725KB/s 1483KB/s 7999KB/s 532K usec 1 4774KB/s 500K usec 8 1610KB/s 621KB/s 9565KB/s 729K usec 1 2910KB/s 677K usec 16 904KB/s 319KB/s 10809KB/s 1431K usec 1 1970KB/s 1399K usec 32 553KB/s 8KB/s 11794KB/s 2330K usec 1 1337KB/s 2398K usec
Notes:
- Decreasing throughput and increasing latencies for sequential reader. Hence no isolation in this case.
- Also note the in case of "32" readers, difference between "max-bw" and "min-bw" is relatively large, considering that all the 32 readers are of same prio. So bw distribution with-in group is not very good. This is the issue of ioprio with-in group I have pointed many times. Ryo is looking into it now.
Sequential Readers vs Multiple Random Readers
=======================================
Ok, because dm-ioband does not provide fairness in case if heavy IO activity is not going in the group, I decided to run a slightly different test case where 16 sequential readers are running in one group and I run increasing number of random readers in other group to see when do I start getting fairness and its effect.
[fio1 --rw=read --bs=4K --size=2G --runtime=30 --direct=1 ]
[fio2 --rw=randread --bs=4K --size=1G --runtime=30 --direct=1 --group_reporting]
Vanilla CFQ
-----------
[Sequential Readers] [Multiple Random Readers]
nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 16 961KB/s 280KB/s 13978KB/s 1673K usec 1 10KB/s 3223 msec 16 903KB/s 260KB/s 12925KB/s 1770K usec 2 28KB/s 3465 msec 16 832KB/s 231KB/s 11428KB/s 2088K usec 4 57KB/s 3891K usec 16 765KB/s 187KB/s 9899KB/s 2500K usec 8 99KB/s 3937K usec 16 512KB/s 144KB/s 6759KB/s 3451K usec 16 148KB/s 5470K usec
IO scheduler controller + CFQ
-----------------------------
[Sequential Readers] [Multiple Random Readers]
nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 16 456KB/s 112KB/s 6380KB/s 3361K usec 1 221KB/s 503K usec 16 476KB/s 159KB/s 6040KB/s 3432K usec 2 214KB/s 549K usec 16 606KB/s 178KB/s 6052KB/s 3801K usec 4 177KB/s 1341K usec 16 589KB/s 83KB/s 6243KB/s 3394K usec 8 154KB/s 3288K usec 16 547KB/s 122KB/s 6122KB/s 3538K usec 16 145KB/s 5959K usec
Notes:
- Stable BW and latencies for sequential reader group in the face of increasing number of random readers in other group.
- Because disk is divided half/half in terms of time, random reader group also gets decent amount of job done. Not sure why BW dips a bit when number of random readers increases. Too seeky to handle?
dm-ioaband + CFQ
----------------
[Sequential Readers] [Multiple Random Readers]
nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 16 926KB/s 293KB/s 10256KB/s 1634K usec 1 55KB/s 1377K usec 16 906KB/s 284KB/s 9240KB/s 1825K usec 2 71KB/s 2392K usec 16 321KB/s 18KB/s 1621KB/s 2037K usec 4 326KB/s 2054K usec 16 188KB/s 16KB/s 1188KB/s 9757K usec 8 404KB/s 3269K usec 16 167KB/s 64KB/s 1700KB/s 2859K usec 16 1064KB/s 2920K usec
Notes:
- Looks like ioband tried to provide fairness from the time when number of random readers are 4. Note, there is sudden increase in BW of random readers and drastic drop in BW of sequential readers.
- By the time number of readers reach 16, total array throughput reduces to around 2.7 MB/s. It got killed because suddenly we are trying to provide fairness in terms of size of IO. That's why on seeky media fairness in terms of disk time works better.
- There is no isolation between groups. Throughput of sequential reader group continues to drop and latencies rise.
- I think these are serious issues which should be looked into and fixed.
Thanks
Vivek
Re: More performance numbers (Was: Re: IO scheduler based IO controller V10)
Quoted 4 lines · Vivek Goyal
Apart from IO scheduler controller number, I also got a chance to run same tests with dm-ioband controller. I am posting these too. I am also planning to run similar numbers on Andrea's "max bw" controller also. Should be able to post those numbers also in 2-3 days.
For those who are interested (expecially to help Vivek to test all this stuff) here the all-in-one patchset of the io-throttle controller, rebased to 2.6.31:
http://www.develer.com/~arighi/linux/patches/io-throttle/old/cgroup-io-throttle-2.6.31.patch
And this one is v18 rebased to 2.6.32-rc3:
http://www.develer.com/~arighi/linux/patches/io-throttle/cgroup-io-throttle-v18.patch
Thanks,
-Andrea
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: More performance numbers (Was: Re: IO scheduler based IO controller V10)
Quoted 4 lines · Vivek Goyal
Apart from IO scheduler controller number, I also got a chance to run same tests with dm-ioband controller. I am posting these too. I am also planning to run similar numbers on Andrea's "max bw" controller also. Should be able to post those numbers also in 2-3 days.
For those who are interested (expecially to help Vivek to test all this stuff) here the all-in-one patchset of the io-throttle controller, rebased to 2.6.31:
http://www.develer.com/~arighi/linux/patches/io-throttle/old/cgroup-io-throttle-2.6.31.patch
And this one is v18 rebased to 2.6.32-rc3:
http://www.develer.com/~arighi/linux/patches/io-throttle/cgroup-io-throttle-v18.patch
Thanks,
-Andrea
Performance numbers with IO throttling patches (Was: Re: IO scheduler based IO controller V10)
In reply to Andrew MortonOn Thu, Sep 24, 2009 at 02:33:15PM -0700, Andrew Morton wrote:
[..]
Quoted 10 lines
Quoted 3 lines
Environment
==========
A 7200 RPM SATA drive with queue depth of 31. Ext3 filesystem.That's a bit of a toy.
Do we have testing results for more enterprisey hardware? Big storage arrays? SSD? Infiniband? iscsi? nfs? (lol, gotcha)
Hi All,
Couple of days back I posted some performance number of "IO scheduler controller" and "dm-ioband" here.
http://lkml.org/lkml/2009/10/8/9
Now I have run similar tests with Andrea Righi's IO throttling approach of max bandwidth control. This is the exercise to understand pros/cons of each approach and see how can we take things forward.
Environment
===========
Software
--------
- 2.6.31 kenrel
- IO scheduler controller V10 on top of 2.6.31
- IO throttling patch on top of 2.6.31. Patch is available here.
http://www.develer.com/~arighi/linux/patches/io-throttle/old/cgroup-io-throttle-2.6.31.patch
Hardware
--------
A storage array of 5 striped disks of 500GB each.
Used fio jobs for 30 seconds in various configurations. Most of the IO is direct IO to eliminate the effects of caches.
I have run three sets for each test. Blindly reporting results of set2 from each test, otherwise it is too much of data to report.
Had lun of 2500GB capacity. Used 200G partition with ext3 file system for my testing. For IO scheduler controller testing, created two cgroups of weight 100 each so that effectively disk can be divided half/half between two groups.
For IO throttling patches also created two cgroups. Now tricky part is that it is a max bw controller and not a proportional weight controller. So dividing the disk capacity half/half between two cgroups is tricky. The reason being I just don't know what's the BW capacity of underlying storage. Throughput varies so much with type of workload. For example, on my arrary, this is how throughput looks like with different workloads.
8 sequential buffered readers 115 MB/s
8 direct sequential readers bs=64K 64 MB/s
8 direct sequential readers bs=4K 14 MB/s
8 buffered random readers bs=64K 3 MB/s
8 direct random readers bs=64K 15 MB/s
8 direct random readers bs=4K 1.5 MB/s
So throughput seems to be varying from 1.5 MB/s to 115 MB/s depending on workload. What should be the BW limits per cgroup to divide disk BW in half/half between two groups?
So I took a conservative estimate and divide max bandwidth divide by 2, and thought of array capacity as 60MB/s and assign each cgroup 30MB/s. In some cases I have assigened even 10MB/s or 5MB/s to each cgropu to see the effects of throttling. I am using "Leaky bucket" policy for all the tests.
As theme of two controllers is different, at some places it might sound like apples vs oranges comparison. But still it does help...
Multiple Random Reader vs Sequential Reader
===============================================
Generally random readers bring the throughput down of others in the system. Ran a test to see the impact of increasing number of random readers on single sequential reader in different groups.
Vanilla CFQ
-----------------------------------
[Multiple Random Reader] [Sequential Reader]
nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 23KB/s 23KB/s 22KB/s 691 msec 1 13519KB/s 468K usec 2 152KB/s 152KB/s 297KB/s 244K usec 1 12380KB/s 31675 usec 4 174KB/s 156KB/s 638KB/s 249K usec 1 10860KB/s 36715 usec 8 49KB/s 11KB/s 310KB/s 1856 msec 1 1292KB/s 990K usec 16 63KB/s 48KB/s 877KB/s 762K usec 1 3905KB/s 506K usec 32 35KB/s 27KB/s 951KB/s 2655 msec 1 1109KB/s 1910K usec
IO scheduler controller + CFQ
-----------------------------------
[Multiple Random Reader] [Sequential Reader]
nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 228KB/s 228KB/s 223KB/s 132K usec 1 5551KB/s 129K usec 2 97KB/s 97KB/s 190KB/s 154K usec 1 5718KB/s 122K usec 4 115KB/s 110KB/s 445KB/s 208K usec 1 5909KB/s 116K usec 8 23KB/s 12KB/s 158KB/s 2820 msec 1 5445KB/s 168K usec 16 11KB/s 3KB/s 145KB/s 5963 msec 1 5418KB/s 164K usec 32 6KB/s 2KB/s 139KB/s 12762 msec 1 5398KB/s 175K usec
Notes:
- Sequential reader in group2 seems to be well isolated from random readers in group1. Throughput and latency of sequential reader are stable and don't drop as number of random readers inrease in system.
io-throttle + CFQ
------------------
BW limit group1=10 MB/s BW limit group2=10 MB/s
[Multiple Random Reader] [Sequential Reader]
nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 37KB/s 37KB/s 36KB/s 218K usec 1 8006KB/s 20529 usec 2 185KB/s 183KB/s 360KB/s 228K usec 1 7475KB/s 33665 usec 4 188KB/s 171KB/s 699KB/s 262K usec 1 6800KB/s 46224 usec 8 84KB/s 51KB/s 573KB/s 1800K usec 1 2835KB/s 885K usec 16 21KB/s 9KB/s 294KB/s 3590 msec 1 437KB/s 1855K usec 32 34KB/s 27KB/s 980KB/s 2861K usec 1 1145KB/s 1952K usec
Notes:
- I have setup limits of 10MB/s in both the cgroups. Now random reader group will never achieve that kind of speed, so it will not be throttled and then it goes onto impact the throughput and latency of other groups in the system.
- Now the key question is how conservative one should in be setting up max BW limit. On this box if a customer has bought 10MB/s cgroup and if he is running some random readers it will kill throughput of other groups in the system and their latencies will shoot up. No isolation in this case.
- So in general, max BW provides isolation from high speed groups but it does not provide isolaton from random reader groups which are moving slow.
Multiple Sequential Reader vs Random Reader
===============================================
Now running a reverse test where in one group I am running increasing number of sequential readers and in other group I am running one random reader and see the impact of sequential readers on random reader.
Vanilla CFQ
-----------------------------------
[Multiple Sequential Reader] [Random Reader]
nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 13978KB/s 13978KB/s 13650KB/s 27614 usec 1 22KB/s 227 msec 2 6225KB/s 6166KB/s 12101KB/s 568K usec 1 10KB/s 457 msec 4 4052KB/s 2462KB/s 13107KB/s 322K usec 1 6KB/s 841 msec 8 1899KB/s 557KB/s 12960KB/s 829K usec 1 13KB/s 1628 msec 16 1007KB/s 279KB/s 13833KB/s 1629K usec 1 10KB/s 3236 msec 32 506KB/s 98KB/s 13704KB/s 3389K usec 1 6KB/s 3238 msec
IO scheduler controller + CFQ
-----------------------------------
[Multiple Sequential Reader] [Random Reader]
nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 5721KB/s 5721KB/s 5587KB/s 126K usec 1 223KB/s 126K usec 2 3216KB/s 1442KB/s 4549KB/s 349K usec 1 224KB/s 176K usec 4 1895KB/s 640KB/s 5121KB/s 775K usec 1 222KB/s 189K usec 8 957KB/s 285KB/s 6368KB/s 1680K usec 1 223KB/s 142K usec 16 458KB/s 132KB/s 6455KB/s 3343K usec 1 219KB/s 165K usec 32 248KB/s 55KB/s 6001KB/s 6957K usec 1 220KB/s 504K usec
Notes:
- Random reader is well isolated from increasing number of sequential readers in other group. BW and latencies are stable.
io-throttle + CFQ
-----------------------------------
BW limit group1=10 MB/s BW limit group2=10 MB/s
[Multiple Sequential Reader] [Random Reader]
nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 8200KB/s 8200KB/s 8007KB/s 20275 usec 1 37KB/s 217K usec 2 3926KB/s 3919KB/s 7661KB/s 122K usec 1 16KB/s 441 msec 4 2271KB/s 1497KB/s 7672KB/s 611K usec 1 9KB/s 927 msec 8 1113KB/s 513KB/s 7507KB/s 849K usec 1 21KB/s 1020 msec 16 661KB/s 236KB/s 7959KB/s 1679K usec 1 13KB/s 2926 msec 32 292KB/s 109KB/s 7864KB/s 3446K usec 1 8KB/s 3439 msec
BW limit group1=5 MB/s BW limit group2=5 MB/s
[Multiple Sequential Reader] [Random Reader]
nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 4686KB/s 4686KB/s 4576KB/s 21095 usec 1 57KB/s 219K usec 2 2298KB/s 2179KB/s 4372KB/s 132K usec 1 37KB/s 431K usec 4 1245KB/s 1019KB/s 4449KB/s 324K usec 1 26KB/s 835 msec 8 584KB/s 403KB/s 4109KB/s 833K usec 1 30KB/s 1625K usec 16 346KB/s 252KB/s 4605KB/s 1641K usec 1 129KB/s 3236K usec 32 175KB/s 56KB/s 4269KB/s 3236K usec 1 8KB/s 3235 msec
Notes:
- Above result is surprising to me. I have run it twice. In first run, I setup per cgroup limit as 10MB/s and in second run I set it up 5MB/s. In both the cases as number of sequential readers increase in other groups, random reader's throughput decreases and latencies increase. This is happening despite the fact that sequential readers are being throttled to make sure it does not impact workload in other group. Wondering why random readers are not seeing consistent throughput and latencies.
- Andrea, can you please also run similar tests to see if you see same results or not. This is to rule out any testing methodology errors or scripting bugs. :-). I also have collected the snapshot of some cgroup files like bandwidth-max, throttlecnt, and stats. Let me know if you want those to see what is happenig here.
Multiple Sequential Reader vs Sequential Reader
===============================================
- This time running random readers are out of the picture and trying to see the effect of increasing number of sequential readers on another sequential reader running in a different group.
Vanilla CFQ
-----------------------------------
[Multiple Sequential Reader] [Sequential Reader]
nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 6325KB/s 6325KB/s 6176KB/s 114K usec 1 6902KB/s 120K usec 2 4588KB/s 3102KB/s 7510KB/s 571K usec 1 4564KB/s 680K usec 4 3242KB/s 1158KB/s 9469KB/s 495K usec 1 3198KB/s 410K usec 8 1775KB/s 459KB/s 12011KB/s 1178K usec 1 1366KB/s 818K usec 16 943KB/s 296KB/s 13285KB/s 1923K usec 1 728KB/s 1816K usec 32 511KB/s 148KB/s 13555KB/s 3286K usec 1 391KB/s 3212K usec
IO scheduler controller + CFQ
-----------------------------------
[Multiple Sequential Reader] [Sequential Reader]
nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 6781KB/s 6781KB/s 6622KB/s 109K usec 1 6691KB/s 115K usec 2 3758KB/s 1876KB/s 5502KB/s 693K usec 1 6373KB/s 419K usec 4 2100KB/s 671KB/s 5751KB/s 987K usec 1 6330KB/s 569K usec 8 1023KB/s 355KB/s 6969KB/s 1569K usec 1 6086KB/s 120K usec 16 520KB/s 130KB/s 7094KB/s 3140K usec 1 5984KB/s 119K usec 32 245KB/s 86KB/s 6621KB/s 6571K usec 1 5850KB/s 113K usec
Notes:
- BW and latencies of sequential reader in group 2 are fairly stable as number of readers increase in first group.
io-throttle + CFQ
-----------------------------------
BW limit group1=30 MB/s BW limit group2=30 MB/s
[Multiple Sequential Reader] [Sequential Reader]
nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 6343KB/s 6343KB/s 6195KB/s 116K usec 1 6993KB/s 109K usec 2 4583KB/s 3046KB/s 7451KB/s 583K usec 1 4516KB/s 433K usec 4 2945KB/s 1324KB/s 9552KB/s 602K usec 1 3001KB/s 583K usec 8 1804KB/s 473KB/s 12257KB/s 861K usec 1 1386KB/s 815K usec 16 942KB/s 265KB/s 13560KB/s 1659K usec 1 718KB/s 1658K usec 32 462KB/s 143KB/s 13757KB/s 3482K usec 1 409KB/s 3480K usec
Notes:
- BW decreases and latencies increase in group2 as number of readers increase in first group. This should be due to fact that no throttling will happen as none of the groups is hitting the limit of 30MB/s. To me this is the tricky part. How a service provider is supposed to set the limit of groups. If groups are not hitting max limits, it will still impact the BW and latencies in other group.
BW limit group1=10 MB/s BW limit group2=10 MB/s
[Multiple Sequential Reader] [Sequential Reader]
nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 4128KB/s 4128KB/s 4032KB/s 215K usec 1 4076KB/s 170K usec 2 2880KB/s 1886KB/s 4655KB/s 291K usec 1 2891KB/s 212K usec 4 1912KB/s 888KB/s 5872KB/s 417K usec 1 1881KB/s 411K usec 8 1032KB/s 432KB/s 7312KB/s 841K usec 1 853KB/s 816K usec 16 540KB/s 259KB/s 7844KB/s 1728K usec 1 503KB/s 1609K usec 32 291KB/s 111KB/s 7920KB/s 3417K usec 1 249KB/s 3205K usec
Notes:
- Same test with 10MB/s as group limit. This is again a surprising result. Max BW in first group is being throttled but still throughput is dropping significantly in second group and latencies are on the rise.
- Limit of first group is 10MB/s but it is achieving max BW of around 8MB/s only. What happened to rest of the 2MB/s?
- Andrea, again, please do run this test. The throughput drop in second group stumps me and forces me to think if I am doing something wrong.
BW limit group1=5 MB/s BW limit group2=5 MB/s
[Multiple Sequential Reader] [Sequential Reader]
nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 2434KB/s 2434KB/s 2377KB/s 110K usec 1 2415KB/s 120K usec 2 1639KB/s 1186KB/s 2759KB/s 222K usec 1 1709KB/s 220K usec 4 1114KB/s 648KB/s 3314KB/s 420K usec 1 1163KB/s 414K usec 8 567KB/s 366KB/s 4060KB/s 901K usec 1 527KB/s 816K usec 16 329KB/s 179KB/s 4324KB/s 1613K usec 1 311KB/s 1613K usec 32 178KB/s 70KB/s 4320KB/s 3235K usec 1 163KB/s 3209K usec
- Setting the limit to 5MB/s per group also does not seem to be helping the second group.
Multiple Random Writer vs Random Reader
===============================================
This time running multiple random writers in first group and see the impact on throughput and latency of random reader in different group.
Vanilla CFQ
-----------------------------------
[Multiple Random Writer] [Random Reader]
nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 64018KB/s 64018KB/s 62517KB/s 353K usec 1 190KB/s 96 msec 2 35298KB/s 35257KB/s 68899KB/s 208K usec 1 76KB/s 2416 msec 4 16387KB/s 14662KB/s 60630KB/s 3746K usec 1 106KB/s 2308K usec 8 5106KB/s 3492KB/s 33335KB/s 2995K usec 1 193KB/s 2292K usec 16 3676KB/s 3002KB/s 51807KB/s 2283K usec 1 72KB/s 2298K usec 32 2169KB/s 1480KB/s 56882KB/s 1990K usec 1 35KB/s 1093 msec
IO scheduler controller + CFQ
-----------------------------------
[Multiple Random Writer] [Random Reader]
nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 20369KB/s 20369KB/s 19892KB/s 877K usec 1 255KB/s 137K usec 2 14347KB/s 14288KB/s 27964KB/s 1010K usec 1 228KB/s 117K usec 4 6996KB/s 6701KB/s 26775KB/s 1362K usec 1 221KB/s 180K usec 8 2849KB/s 2770KB/s 22007KB/s 2660K usec 1 250KB/s 485K usec 16 1463KB/s 1365KB/s 22384KB/s 2606K usec 1 254KB/s 115K usec 32 799KB/s 681KB/s 22404KB/s 2879K usec 1 266KB/s 107K usec
Notes
- BW and latencies of random reader in second group are fairly stable.
io-throttle + CFQ
-----------------------------------
BW limit group1=30 MB/s BW limit group2=30 MB/s
[Multiple Random Writer] [Random Reader]
nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 21920KB/s 21920KB/s 21406KB/s 1017K usec 1 353KB/s 432K usec 2 14291KB/s 9626KB/s 23357KB/s 1832K usec 1 362KB/s 177K usec 4 7130KB/s 5135KB/s 24736KB/s 1336K usec 1 348KB/s 425K usec 8 3165KB/s 2949KB/s 23792KB/s 2133K usec 1 336KB/s 146K usec 16 1653KB/s 1406KB/s 23694KB/s 2198K usec 1 337KB/s 115K usec 32 793KB/s 717KB/s 23198KB/s 2195K usec 1 330KB/s 192K usec
BW limit group1=10 MB/s BW limit group2=10 MB/s
[Multiple Random Writer] [Random Reader]
nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 7903KB/s 7903KB/s 7718KB/s 1037K usec 1 474KB/s 103K usec 2 4496KB/s 4428KB/s 8715KB/s 1091K usec 1 450KB/s 553K usec 4 2153KB/s 1827KB/s 7914KB/s 2042K usec 1 458KB/s 108K usec 8 1129KB/s 1087KB/s 8688KB/s 1280K usec 1 432KB/s 98215 usec 16 606KB/s 527KB/s 8668KB/s 2303K usec 1 426KB/s 90609 usec 32 312KB/s 259KB/s 8599KB/s 2557K usec 1 441KB/s 95283 usec
Notes:
- IO throttling seems to be working really well here. Random writers are contained in the first group and this gives stable BW and latencies to random reader in second group.
Multiple Buffered Writer vs Buffered Writer
===========================================
This time run multiple buffered writers in group1 and see run a single buffered writer in other group and see if we can provide fairness and isolation.
Vanilla CFQ
------------
[Multiple Buffered Writer] [Buffered Writer]
nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 68997KB/s 68997KB/s 67380KB/s 645K usec 1 67122KB/s 567K usec 2 47509KB/s 46218KB/s 91510KB/s 865K usec 1 45118KB/s 865K usec 4 28002KB/s 26906KB/s 105MB/s 1649K usec 1 26879KB/s 1643K usec 8 15985KB/s 14849KB/s 117MB/s 943K usec 1 15653KB/s 766K usec 16 11567KB/s 6881KB/s 128MB/s 1174K usec 1 7333KB/s 947K usec 32 5877KB/s 3649KB/s 130MB/s 1205K usec 1 5142KB/s 988K usec
IO scheduler controller + CFQ
-----------------------------------
[Multiple Buffered Writer] [Buffered Writer]
nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 68580KB/s 68580KB/s 66972KB/s 2901K usec 1 67194KB/s 2901K usec 2 47419KB/s 45700KB/s 90936KB/s 3149K usec 1 44628KB/s 2377K usec 4 27825KB/s 27274KB/s 105MB/s 1177K usec 1 27584KB/s 1177K usec 8 15382KB/s 14288KB/s 114MB/s 1539K usec 1 14794KB/s 783K usec 16 9161KB/s 7592KB/s 124MB/s 3177K usec 1 7713KB/s 886K usec 32 4928KB/s 3961KB/s 126MB/s 1152K usec 1 6465KB/s 4510K usec
Notes:
- It does not work. Buffered writer in second group are being overwhelmed by writers in group1.
- This is a limitation of IO scheduler based controller currently as page cache at higher layer evens out the traffic and does not throw more traffic from higher weight group.
- This is something needs more work at higher layers like dirty limts per cgroup in memory contoller and the method to writeout buffered pages belonging to a particular memory cgroup. This is still being brainstormed.
io-throttle + CFQ
-----------------------------------
BW limit group1=30 MB/s BW limit group2=30 MB/s
[Multiple Buffered Writer] [Buffered Writer]
nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 33863KB/s 33863KB/s 33070KB/s 3046K usec 1 25165KB/s 13248K usec 2 13457KB/s 12906KB/s 25745KB/s 9286K usec 1 29958KB/s 3736K usec 4 7414KB/s 6543KB/s 27145KB/s 10557K usec 1 30968KB/s 8356K usec 8 3562KB/s 2640KB/s 24430KB/s 12012K usec 1 30801KB/s 7037K usec 16 3962KB/s 881KB/s 26632KB/s 12650K usec 1 31150KB/s 7173K usec 32 3275KB/s 406KB/s 27295KB/s 14609K usec 1 26328KB/s 8069K usec
Notes:
- This seems to work well here. io-throttle is throttling the writers before they write too much of data in page cache. One side effect of this seems to be that now a process will not be allowed to write at memory speed in page cahce and will be limited to disk IO speed limits set for the cgroup.
Andrea is thinking of removing throttling in balance_dirty_pages() to allow writting at disk speed till we hit dirty_limits. But removing it leads to a different issue where too many dirty pages from a single group can be present from a cgroup in page cache and if that cgroup is slow moving one, then pages are flushed to disk at slower speed delyaing other higher rate cgroups. (all discussed in private mails with Andrea).
ioprio class and iopriority with-in cgroups issues with IO-throttle
===================================================================
Currently throttling logic is designed in such a way that it makes the throttling uniform for every process in the group. So we will loose the differentiation between different class of processes or differnetitation between different priority of processes with-in group.
I have run the tests of these in the past and reported it here in the past.
https://lists.linux-foundation.org/pipermail/containers/2009-May/017588.html
Thanks
Vivek
Re: Performance numbers with IO throttling patches (Was: Re: IO scheduler based IO controller V10)
Quoted 7 lines · Vivek Goyal
On Thu, Sep 24, 2009 at 02:33:15PM -0700, Andrew Morton wrote:
[..]
Quoted 4 lines
Quoted 3 lines
Environment
==========
A 7200 RPM SATA drive with queue depth of 31. Ext3 filesystem.
Quoted 8 lines
Quoted 6 lines
That's a bit of a toy.
Do we have testing results for more enterprisey hardware? Big storage arrays? SSD? Infiniband? iscsi? nfs? (lol, gotcha)
Hi All,
Hi Vivek,
thanks for posting this detailed report first of all. A few comments below.
Quoted 119 lines
Couple of days back I posted some performance number of "IO scheduler controller" and "dm-ioband" here.
http://lkml.org/lkml/2009/10/8/9
Now I have run similar tests with Andrea Righi's IO throttling approach of max bandwidth control. This is the exercise to understand pros/cons of each approach and see how can we take things forward.
Environment
===========
Software
--------
- 2.6.31 kenrel
- IO scheduler controller V10 on top of 2.6.31
- IO throttling patch on top of 2.6.31. Patch is available here.http://www.develer.com/~arighi/linux/patches/io-throttle/old/cgroup-io-throttle-2.6.31.patch
Hardware
--------
A storage array of 5 striped disks of 500GB each.Used fio jobs for 30 seconds in various configurations. Most of the IO is direct IO to eliminate the effects of caches.
I have run three sets for each test. Blindly reporting results of set2 from each test, otherwise it is too much of data to report.
Had lun of 2500GB capacity. Used 200G partition with ext3 file system for my testing. For IO scheduler controller testing, created two cgroups of weight 100 each so that effectively disk can be divided half/half between two groups.
For IO throttling patches also created two cgroups. Now tricky part is that it is a max bw controller and not a proportional weight controller. So dividing the disk capacity half/half between two cgroups is tricky. The reason being I just don't know what's the BW capacity of underlying storage. Throughput varies so much with type of workload. For example, on my arrary, this is how throughput looks like with different workloads.
8 sequential buffered readers 115 MB/s
8 direct sequential readers bs=64K 64 MB/s
8 direct sequential readers bs=4K 14 MB/s8 buffered random readers bs=64K 3 MB/s
8 direct random readers bs=64K 15 MB/s
8 direct random readers bs=4K 1.5 MB/sSo throughput seems to be varying from 1.5 MB/s to 115 MB/s depending on workload. What should be the BW limits per cgroup to divide disk BW in half/half between two groups?
So I took a conservative estimate and divide max bandwidth divide by 2, and thought of array capacity as 60MB/s and assign each cgroup 30MB/s. In some cases I have assigened even 10MB/s or 5MB/s to each cgropu to see the effects of throttling. I am using "Leaky bucket" policy for all the tests.
As theme of two controllers is different, at some places it might sound like apples vs oranges comparison. But still it does help...
Multiple Random Reader vs Sequential Reader
===============================================
Generally random readers bring the throughput down of others in the system. Ran a test to see the impact of increasing number of random readers on single sequential reader in different groups.Vanilla CFQ
-----------------------------------
[Multiple Random Reader] [Sequential Reader]nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 23KB/s 23KB/s 22KB/s 691 msec 1 13519KB/s 468K usec 2 152KB/s 152KB/s 297KB/s 244K usec 1 12380KB/s 31675 usec 4 174KB/s 156KB/s 638KB/s 249K usec 1 10860KB/s 36715 usec 8 49KB/s 11KB/s 310KB/s 1856 msec 1 1292KB/s 990K usec 16 63KB/s 48KB/s 877KB/s 762K usec 1 3905KB/s 506K usec 32 35KB/s 27KB/s 951KB/s 2655 msec 1 1109KB/s 1910K usecIO scheduler controller + CFQ
-----------------------------------
[Multiple Random Reader] [Sequential Reader]nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 228KB/s 228KB/s 223KB/s 132K usec 1 5551KB/s 129K usec 2 97KB/s 97KB/s 190KB/s 154K usec 1 5718KB/s 122K usec 4 115KB/s 110KB/s 445KB/s 208K usec 1 5909KB/s 116K usec 8 23KB/s 12KB/s 158KB/s 2820 msec 1 5445KB/s 168K usec 16 11KB/s 3KB/s 145KB/s 5963 msec 1 5418KB/s 164K usec 32 6KB/s 2KB/s 139KB/s 12762 msec 1 5398KB/s 175K usecNotes:
- Sequential reader in group2 seems to be well isolated from random readers in group1. Throughput and latency of sequential reader are stable and don't drop as number of random readers inrease in system.io-throttle + CFQ
------------------
BW limit group1=10 MB/s BW limit group2=10 MB/s
[Multiple Random Reader] [Sequential Reader]nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 37KB/s 37KB/s 36KB/s 218K usec 1 8006KB/s 20529 usec 2 185KB/s 183KB/s 360KB/s 228K usec 1 7475KB/s 33665 usec 4 188KB/s 171KB/s 699KB/s 262K usec 1 6800KB/s 46224 usec 8 84KB/s 51KB/s 573KB/s 1800K usec 1 2835KB/s 885K usec 16 21KB/s 9KB/s 294KB/s 3590 msec 1 437KB/s 1855K usec 32 34KB/s 27KB/s 980KB/s 2861K usec 1 1145KB/s 1952K usecNotes:
- I have setup limits of 10MB/s in both the cgroups. Now random reader group will never achieve that kind of speed, so it will not be throttled and then it goes onto impact the throughput and latency of other groups in the system.- Now the key question is how conservative one should in be setting up max BW limit. On this box if a customer has bought 10MB/s cgroup and if he is running some random readers it will kill throughput of other groups in the system and their latencies will shoot up. No isolation in this case.
Quoted 3 lines
- So in general, max BW provides isolation from high speed groups but it does not provide isolaton from random reader groups which are moving slow.
Remember that in addition to blockio.bandwidth-max the io-throttle controlller also provides blockio.iops-max to enforce hard limits on the number of IO operations per second. Probably for this testcase both cgroups should be limited in terms of BW and iops to achieve a better isolation.
Quoted 57 lines
Multiple Sequential Reader vs Random Reader
===============================================
Now running a reverse test where in one group I am running increasing number of sequential readers and in other group I am running one random reader and see the impact of sequential readers on random reader.Vanilla CFQ
-----------------------------------
[Multiple Sequential Reader] [Random Reader]nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 13978KB/s 13978KB/s 13650KB/s 27614 usec 1 22KB/s 227 msec 2 6225KB/s 6166KB/s 12101KB/s 568K usec 1 10KB/s 457 msec 4 4052KB/s 2462KB/s 13107KB/s 322K usec 1 6KB/s 841 msec 8 1899KB/s 557KB/s 12960KB/s 829K usec 1 13KB/s 1628 msec 16 1007KB/s 279KB/s 13833KB/s 1629K usec 1 10KB/s 3236 msec 32 506KB/s 98KB/s 13704KB/s 3389K usec 1 6KB/s 3238 msecIO scheduler controller + CFQ
-----------------------------------
[Multiple Sequential Reader] [Random Reader]nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 5721KB/s 5721KB/s 5587KB/s 126K usec 1 223KB/s 126K usec 2 3216KB/s 1442KB/s 4549KB/s 349K usec 1 224KB/s 176K usec 4 1895KB/s 640KB/s 5121KB/s 775K usec 1 222KB/s 189K usec 8 957KB/s 285KB/s 6368KB/s 1680K usec 1 223KB/s 142K usec 16 458KB/s 132KB/s 6455KB/s 3343K usec 1 219KB/s 165K usec 32 248KB/s 55KB/s 6001KB/s 6957K usec 1 220KB/s 504K usecNotes:
- Random reader is well isolated from increasing number of sequential readers in other group. BW and latencies are stable.io-throttle + CFQ
-----------------------------------
BW limit group1=10 MB/s BW limit group2=10 MB/s
[Multiple Sequential Reader] [Random Reader]nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 8200KB/s 8200KB/s 8007KB/s 20275 usec 1 37KB/s 217K usec 2 3926KB/s 3919KB/s 7661KB/s 122K usec 1 16KB/s 441 msec 4 2271KB/s 1497KB/s 7672KB/s 611K usec 1 9KB/s 927 msec 8 1113KB/s 513KB/s 7507KB/s 849K usec 1 21KB/s 1020 msec 16 661KB/s 236KB/s 7959KB/s 1679K usec 1 13KB/s 2926 msec 32 292KB/s 109KB/s 7864KB/s 3446K usec 1 8KB/s 3439 msecBW limit group1=5 MB/s BW limit group2=5 MB/s
[Multiple Sequential Reader] [Random Reader]nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 4686KB/s 4686KB/s 4576KB/s 21095 usec 1 57KB/s 219K usec 2 2298KB/s 2179KB/s 4372KB/s 132K usec 1 37KB/s 431K usec 4 1245KB/s 1019KB/s 4449KB/s 324K usec 1 26KB/s 835 msec 8 584KB/s 403KB/s 4109KB/s 833K usec 1 30KB/s 1625K usec 16 346KB/s 252KB/s 4605KB/s 1641K usec 1 129KB/s 3236K usec 32 175KB/s 56KB/s 4269KB/s 3236K usec 1 8KB/s 3235 msecNotes:
Quoted 7 lines
- Above result is surprising to me. I have run it twice. In first run, I setup per cgroup limit as 10MB/s and in second run I set it up 5MB/s. In both the cases as number of sequential readers increase in other groups, random reader's throughput decreases and latencies increase. This is happening despite the fact that sequential readers are being throttled to make sure it does not impact workload in other group. Wondering why random readers are not seeing consistent throughput and latencies.
Maybe because CFQ is still trying to be fair among processes instead of cgroups. Remember that io-throttle doesn't touch the CFQ code (for this I'm definitely convinced that CFQ should be changed to think also in terms of cgroups, and io-throttle alone is not enough).
So, even if group1 is being throttled in part it is still able to submit some requests that get a higher priority respect to the requests submitted by the single random reader task.
It could be interesting to test another IO scheduler (deadline, as or even noop) to check if this is the actual problem.
Quoted 6 lines
- Andrea, can you please also run similar tests to see if you see same results or not. This is to rule out any testing methodology errors or scripting bugs. :-). I also have collected the snapshot of some cgroup files like bandwidth-max, throttlecnt, and stats. Let me know if you want those to see what is happenig here.
Sure, I'll do some tests ASAP. Another interesting test would be to set a blockio.iops-max limit also for the sequential readers' cgroup, to be sure we're not touching some iops physical disk limit.
Could you post all the options you used with fio, so I can repeat some tests as similar as possible to yours?
Quoted 45 lines
Multiple Sequential Reader vs Sequential Reader
===============================================
- This time running random readers are out of the picture and trying to see the effect of increasing number of sequential readers on another sequential reader running in a different group.Vanilla CFQ
-----------------------------------
[Multiple Sequential Reader] [Sequential Reader]nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 6325KB/s 6325KB/s 6176KB/s 114K usec 1 6902KB/s 120K usec 2 4588KB/s 3102KB/s 7510KB/s 571K usec 1 4564KB/s 680K usec 4 3242KB/s 1158KB/s 9469KB/s 495K usec 1 3198KB/s 410K usec 8 1775KB/s 459KB/s 12011KB/s 1178K usec 1 1366KB/s 818K usec 16 943KB/s 296KB/s 13285KB/s 1923K usec 1 728KB/s 1816K usec 32 511KB/s 148KB/s 13555KB/s 3286K usec 1 391KB/s 3212K usecIO scheduler controller + CFQ
-----------------------------------
[Multiple Sequential Reader] [Sequential Reader]nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 6781KB/s 6781KB/s 6622KB/s 109K usec 1 6691KB/s 115K usec 2 3758KB/s 1876KB/s 5502KB/s 693K usec 1 6373KB/s 419K usec 4 2100KB/s 671KB/s 5751KB/s 987K usec 1 6330KB/s 569K usec 8 1023KB/s 355KB/s 6969KB/s 1569K usec 1 6086KB/s 120K usec 16 520KB/s 130KB/s 7094KB/s 3140K usec 1 5984KB/s 119K usec 32 245KB/s 86KB/s 6621KB/s 6571K usec 1 5850KB/s 113K usecNotes:
- BW and latencies of sequential reader in group 2 are fairly stable as number of readers increase in first group.io-throttle + CFQ
-----------------------------------
BW limit group1=30 MB/s BW limit group2=30 MB/s
[Multiple Sequential Reader] [Sequential Reader]nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 6343KB/s 6343KB/s 6195KB/s 116K usec 1 6993KB/s 109K usec 2 4583KB/s 3046KB/s 7451KB/s 583K usec 1 4516KB/s 433K usec 4 2945KB/s 1324KB/s 9552KB/s 602K usec 1 3001KB/s 583K usec 8 1804KB/s 473KB/s 12257KB/s 861K usec 1 1386KB/s 815K usec 16 942KB/s 265KB/s 13560KB/s 1659K usec 1 718KB/s 1658K usec 32 462KB/s 143KB/s 13757KB/s 3482K usec 1 409KB/s 3480K usec
Quoted 7 lines
Notes:
- BW decreases and latencies increase in group2 as number of readers increase in first group. This should be due to fact that no throttling will happen as none of the groups is hitting the limit of 30MB/s. To me this is the tricky part. How a service provider is supposed to set the limit of groups. If groups are not hitting max limits, it will still impact the BW and latencies in other group.
Are you using 4k block size here? because in case of too small blocks you could hit some physical iops limit. Also for this case it could be interesting to see what happens setting both BW and iops hard limits.
Quoted 11 lines
BW limit group1=10 MB/s BW limit group2=10 MB/s
[Multiple Sequential Reader] [Sequential Reader]nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 4128KB/s 4128KB/s 4032KB/s 215K usec 1 4076KB/s 170K usec 2 2880KB/s 1886KB/s 4655KB/s 291K usec 1 2891KB/s 212K usec 4 1912KB/s 888KB/s 5872KB/s 417K usec 1 1881KB/s 411K usec 8 1032KB/s 432KB/s 7312KB/s 841K usec 1 853KB/s 816K usec 16 540KB/s 259KB/s 7844KB/s 1728K usec 1 503KB/s 1609K usec 32 291KB/s 111KB/s 7920KB/s 3417K usec 1 249KB/s 3205K usec
Quoted 4 lines
Notes:
- Same test with 10MB/s as group limit. This is again a surprising result. Max BW in first group is being throttled but still throughput is dropping significantly in second group and latencies are on the rise.
Same consideration about CFQ and/or iops limit. Could you post all the fio options you've used also for this test (or better, for all tests)?
Quoted 3 lines
- Limit of first group is 10MB/s but it is achieving max BW of around 8MB/s only. What happened to rest of the 2MB/s?
Ditto.
Quoted 134 lines
- Andrea, again, please do run this test. The throughput drop in second group stumps me and forces me to think if I am doing something wrong.
BW limit group1=5 MB/s BW limit group2=5 MB/s
[Multiple Sequential Reader] [Sequential Reader]nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 2434KB/s 2434KB/s 2377KB/s 110K usec 1 2415KB/s 120K usec 2 1639KB/s 1186KB/s 2759KB/s 222K usec 1 1709KB/s 220K usec 4 1114KB/s 648KB/s 3314KB/s 420K usec 1 1163KB/s 414K usec 8 567KB/s 366KB/s 4060KB/s 901K usec 1 527KB/s 816K usec 16 329KB/s 179KB/s 4324KB/s 1613K usec 1 311KB/s 1613K usec 32 178KB/s 70KB/s 4320KB/s 3235K usec 1 163KB/s 3209K usec- Setting the limit to 5MB/s per group also does not seem to be helping the second group.
Multiple Random Writer vs Random Reader
===============================================
This time running multiple random writers in first group and see the impact on throughput and latency of random reader in different group.Vanilla CFQ
-----------------------------------
[Multiple Random Writer] [Random Reader]nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 64018KB/s 64018KB/s 62517KB/s 353K usec 1 190KB/s 96 msec 2 35298KB/s 35257KB/s 68899KB/s 208K usec 1 76KB/s 2416 msec 4 16387KB/s 14662KB/s 60630KB/s 3746K usec 1 106KB/s 2308K usec 8 5106KB/s 3492KB/s 33335KB/s 2995K usec 1 193KB/s 2292K usec 16 3676KB/s 3002KB/s 51807KB/s 2283K usec 1 72KB/s 2298K usec 32 2169KB/s 1480KB/s 56882KB/s 1990K usec 1 35KB/s 1093 msecIO scheduler controller + CFQ
-----------------------------------
[Multiple Random Writer] [Random Reader]nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 20369KB/s 20369KB/s 19892KB/s 877K usec 1 255KB/s 137K usec 2 14347KB/s 14288KB/s 27964KB/s 1010K usec 1 228KB/s 117K usec 4 6996KB/s 6701KB/s 26775KB/s 1362K usec 1 221KB/s 180K usec 8 2849KB/s 2770KB/s 22007KB/s 2660K usec 1 250KB/s 485K usec 16 1463KB/s 1365KB/s 22384KB/s 2606K usec 1 254KB/s 115K usec 32 799KB/s 681KB/s 22404KB/s 2879K usec 1 266KB/s 107K usecNotes
- BW and latencies of random reader in second group are fairly stable.io-throttle + CFQ
-----------------------------------
BW limit group1=30 MB/s BW limit group2=30 MB/s
[Multiple Random Writer] [Random Reader]nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 21920KB/s 21920KB/s 21406KB/s 1017K usec 1 353KB/s 432K usec 2 14291KB/s 9626KB/s 23357KB/s 1832K usec 1 362KB/s 177K usec 4 7130KB/s 5135KB/s 24736KB/s 1336K usec 1 348KB/s 425K usec 8 3165KB/s 2949KB/s 23792KB/s 2133K usec 1 336KB/s 146K usec 16 1653KB/s 1406KB/s 23694KB/s 2198K usec 1 337KB/s 115K usec 32 793KB/s 717KB/s 23198KB/s 2195K usec 1 330KB/s 192K usecBW limit group1=10 MB/s BW limit group2=10 MB/s
[Multiple Random Writer] [Random Reader]nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 7903KB/s 7903KB/s 7718KB/s 1037K usec 1 474KB/s 103K usec 2 4496KB/s 4428KB/s 8715KB/s 1091K usec 1 450KB/s 553K usec 4 2153KB/s 1827KB/s 7914KB/s 2042K usec 1 458KB/s 108K usec 8 1129KB/s 1087KB/s 8688KB/s 1280K usec 1 432KB/s 98215 usec 16 606KB/s 527KB/s 8668KB/s 2303K usec 1 426KB/s 90609 usec 32 312KB/s 259KB/s 8599KB/s 2557K usec 1 441KB/s 95283 usecNotes:
- IO throttling seems to be working really well here. Random writers are contained in the first group and this gives stable BW and latencies to random reader in second group.Multiple Buffered Writer vs Buffered Writer
===========================================
This time run multiple buffered writers in group1 and see run a single buffered writer in other group and see if we can provide fairness and isolation.Vanilla CFQ
------------
[Multiple Buffered Writer] [Buffered Writer]nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 68997KB/s 68997KB/s 67380KB/s 645K usec 1 67122KB/s 567K usec 2 47509KB/s 46218KB/s 91510KB/s 865K usec 1 45118KB/s 865K usec 4 28002KB/s 26906KB/s 105MB/s 1649K usec 1 26879KB/s 1643K usec 8 15985KB/s 14849KB/s 117MB/s 943K usec 1 15653KB/s 766K usec 16 11567KB/s 6881KB/s 128MB/s 1174K usec 1 7333KB/s 947K usec 32 5877KB/s 3649KB/s 130MB/s 1205K usec 1 5142KB/s 988K usecIO scheduler controller + CFQ
-----------------------------------
[Multiple Buffered Writer] [Buffered Writer]nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 68580KB/s 68580KB/s 66972KB/s 2901K usec 1 67194KB/s 2901K usec 2 47419KB/s 45700KB/s 90936KB/s 3149K usec 1 44628KB/s 2377K usec 4 27825KB/s 27274KB/s 105MB/s 1177K usec 1 27584KB/s 1177K usec 8 15382KB/s 14288KB/s 114MB/s 1539K usec 1 14794KB/s 783K usec 16 9161KB/s 7592KB/s 124MB/s 3177K usec 1 7713KB/s 886K usec 32 4928KB/s 3961KB/s 126MB/s 1152K usec 1 6465KB/s 4510K usecNotes:
- It does not work. Buffered writer in second group are being overwhelmed by writers in group1.- This is a limitation of IO scheduler based controller currently as page cache at higher layer evens out the traffic and does not throw more traffic from higher weight group.
- This is something needs more work at higher layers like dirty limts per cgroup in memory contoller and the method to writeout buffered pages belonging to a particular memory cgroup. This is still being brainstormed.
io-throttle + CFQ
-----------------------------------
BW limit group1=30 MB/s BW limit group2=30 MB/s
[Multiple Buffered Writer] [Buffered Writer]nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 33863KB/s 33863KB/s 33070KB/s 3046K usec 1 25165KB/s 13248K usec 2 13457KB/s 12906KB/s 25745KB/s 9286K usec 1 29958KB/s 3736K usec 4 7414KB/s 6543KB/s 27145KB/s 10557K usec 1 30968KB/s 8356K usec 8 3562KB/s 2640KB/s 24430KB/s 12012K usec 1 30801KB/s 7037K usec 16 3962KB/s 881KB/s 26632KB/s 12650K usec 1 31150KB/s 7173K usec 32 3275KB/s 406KB/s 27295KB/s 14609K usec 1 26328KB/s 8069K usecNotes:
- This seems to work well here. io-throttle is throttling the writers before they write too much of data in page cache. One side effect of this seems to be that now a process will not be allowed to write at memory speed in page cahce and will be limited to disk IO speed limits set for the cgroup.
Quoted 6 lines
Andrea is thinking of removing throttling in
balance_dirty_pages()to allow writting at disk speed till we hitdirty_limits. But removing it leads to a different issue where too many dirty pages from a single group can be present from a cgroup in page cache and if that cgroup is slow moving one, then pages are flushed to disk at slower speed delyaing other higher rate cgroups. (all discussed in private mails with Andrea).
I confirm this. :) But IMHO before removing the throttling in balance_dirty_pages() we really need the per-cgroup dirty limit / dirty page cache quota.
Quoted 17 lines
ioprio class and iopriority with-in cgroups issues with IO-throttle
===================================================================Currently throttling logic is designed in such a way that it makes the throttling uniform for every process in the group. So we will loose the differentiation between different class of processes or differnetitation between different priority of processes with-in group.
I have run the tests of these in the past and reported it here in the past.
https://lists.linux-foundation.org/pipermail/containers/2009-May/017588.html
Thanks
Vivek
Signature · 2 lines
Andrea Righi - Develer s.r.l http://www.develer.com
Re: Performance numbers with IO throttling patches (Was: Re: IO scheduler based IO controller V10)
On Sun, Oct 11, 2009 at 12:27:30AM +0200, Andrea Righi wrote:
[..]
Quoted 7 lines
Quoted 6 lines
- Andrea, can you please also run similar tests to see if you see same results or not. This is to rule out any testing methodology errors or scripting bugs. :-). I also have collected the snapshot of some cgroup files like bandwidth-max, throttlecnt, and stats. Let me know if you want those to see what is happenig here.
Quoted 7 lines
Sure, I'll do some tests ASAP. Another interesting test would be to set a blockio.iops-max limit also for the sequential readers' cgroup, to be sure we're not touching some iops physical disk limit.
Could you post all the options you used with fio, so I can repeat some tests as similar as possible to yours?
I will respond to rest of the points later after some testing with iops-max rules. In the mean time here are my fio options so that you can try to replicate the tests.
I am simply copying pasting from my script. I have written my own program "semwait" so that two different instances of fio can synchronize on an external semaphore. Generally all the jobs go in single fio files but here we need to put two fio instances in two different cgroups. It is important that two fio jobs are synchronized and start at the same time after laying out files. (Becomes primarilly useful in write testing. Reads are fine generally onces the files have been laid out).
Sequential readers
------------------
fio_args="--rw=read --bs=4K --size=2G --runtime=30 --numjobs=$nr_jobs --direct=1"
fio $fio_args --name=$jobname --directory=/mnt/$blockdev/fio --exec_prerun="'/usr/local/bin/semwait fiocgroup'" >> $outputdir/$outputfile &
Random Reader
-------------
fio_args="--rw=randread --bs=4K --size=1G --runtime=30 --direct=1 --numjobs=$nr_jobs"
fio $fio_args --name=$jobname --directory=/mnt/$blockdev/fio --exec_prerun="'/usr/local/bin/semwait fiocgroup'" >> $outputdir/$outputfile &
Random Writer
-------------
fio_args="--rw=randwrite --bs=64K --size=2G --runtime=30 --numjobs=$nr_jobs1 --ioengine=libaio --iodepth=4 --direct=1"
fio $fio_args --name=$jobname --directory=/mnt/$blockdev/fio --exec_prerun="'/usr/local/bin/semwait fiocgroup'" >> $outputdir/$outputfile &
Thanks
Vivek
Re: Performance numbers with IO throttling patches (Was: Re: IO scheduler based IO controller V10)
In reply to Andrea RighiOn Sun, Oct 11, 2009 at 12:27:30AM +0200, Andrea Righi wrote:
[..]
Quoted 61 lines
Quoted 57 lines
Multiple Random Reader vs Sequential Reader
===============================================
Generally random readers bring the throughput down of others in the system. Ran a test to see the impact of increasing number of random readers on single sequential reader in different groups.Vanilla CFQ
-----------------------------------
[Multiple Random Reader] [Sequential Reader]nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 23KB/s 23KB/s 22KB/s 691 msec 1 13519KB/s 468K usec 2 152KB/s 152KB/s 297KB/s 244K usec 1 12380KB/s 31675 usec 4 174KB/s 156KB/s 638KB/s 249K usec 1 10860KB/s 36715 usec 8 49KB/s 11KB/s 310KB/s 1856 msec 1 1292KB/s 990K usec 16 63KB/s 48KB/s 877KB/s 762K usec 1 3905KB/s 506K usec 32 35KB/s 27KB/s 951KB/s 2655 msec 1 1109KB/s 1910K usecIO scheduler controller + CFQ
-----------------------------------
[Multiple Random Reader] [Sequential Reader]nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 228KB/s 228KB/s 223KB/s 132K usec 1 5551KB/s 129K usec 2 97KB/s 97KB/s 190KB/s 154K usec 1 5718KB/s 122K usec 4 115KB/s 110KB/s 445KB/s 208K usec 1 5909KB/s 116K usec 8 23KB/s 12KB/s 158KB/s 2820 msec 1 5445KB/s 168K usec 16 11KB/s 3KB/s 145KB/s 5963 msec 1 5418KB/s 164K usec 32 6KB/s 2KB/s 139KB/s 12762 msec 1 5398KB/s 175K usecNotes:
- Sequential reader in group2 seems to be well isolated from random readers in group1. Throughput and latency of sequential reader are stable and don't drop as number of random readers inrease in system.io-throttle + CFQ
------------------
BW limit group1=10 MB/s BW limit group2=10 MB/s
[Multiple Random Reader] [Sequential Reader]nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 37KB/s 37KB/s 36KB/s 218K usec 1 8006KB/s 20529 usec 2 185KB/s 183KB/s 360KB/s 228K usec 1 7475KB/s 33665 usec 4 188KB/s 171KB/s 699KB/s 262K usec 1 6800KB/s 46224 usec 8 84KB/s 51KB/s 573KB/s 1800K usec 1 2835KB/s 885K usec 16 21KB/s 9KB/s 294KB/s 3590 msec 1 437KB/s 1855K usec 32 34KB/s 27KB/s 980KB/s 2861K usec 1 1145KB/s 1952K usecNotes:
- I have setup limits of 10MB/s in both the cgroups. Now random reader group will never achieve that kind of speed, so it will not be throttled and then it goes onto impact the throughput and latency of other groups in the system.- Now the key question is how conservative one should in be setting up max BW limit. On this box if a customer has bought 10MB/s cgroup and if he is running some random readers it will kill throughput of other groups in the system and their latencies will shoot up. No isolation in this case.
Quoted 3 lines
- So in general, max BW provides isolation from high speed groups but it does not provide isolaton from random reader groups which are moving slow.
Quoted 6 lines
Remember that in addition to blockio.bandwidth-max the io-throttle controlller also provides blockio.iops-max to enforce hard limits on the number of IO operations per second. Probably for this testcase both cgroups should be limited in terms of BW and iops to achieve a better isolation.
I modified my report scripts to also output aggreagate iops numbers and remove max-bandwidth and min-bandwidth numbers. So for same tests and same results I am now reporting iops numbers also. ( I have not re-run the tests.)
IO scheduler controller + CFQ
-----------------------------------
[Multiple Random Reader] [Sequential Reader] nr Agg-bandw Max-latency Agg-iops nr Agg-bandw Max-latency Agg-iops 1 223KB/s 132K usec 55 1 5551KB/s 129K usec 1387 2 190KB/s 154K usec 46 1 5718KB/s 122K usec 1429 4 445KB/s 208K usec 111 1 5909KB/s 116K usec 1477 8 158KB/s 2820 msec 36 1 5445KB/s 168K usec 1361 16 145KB/s 5963 msec 28 1 5418KB/s 164K usec 1354 32 139KB/s 12762 msec 23 1 5398KB/s 175K usec 1349
io-throttle + CFQ
-----------------------------------
BW limit group1=10 MB/s BW limit group2=10 MB/s [Multiple Random Reader] [Sequential Reader] nr Agg-bandw Max-latency Agg-iops nr Agg-bandw Max-latency Agg-iops 1 36KB/s 218K usec 9 1 8006KB/s 20529 usec 2001 2 360KB/s 228K usec 89 1 7475KB/s 33665 usec 1868 4 699KB/s 262K usec 173 1 6800KB/s 46224 usec 1700 8 573KB/s 1800K usec 139 1 2835KB/s 885K usec 708 16 294KB/s 3590 msec 68 1 437KB/s 1855K usec 109 32 980KB/s 2861K usec 230 1 1145KB/s 1952K usec 286
Note that in case of random reader groups, iops are really small. Few thougts.
- What should be the iops limit I should choose for the group. Lets say if I choose "80", then things should be better for sequential reader group, but just think of what will happen to random reader group. Especially, if nature of workload in group1 changes to sequential. Group1 will simply be killed.
So yes, one can limit a group both by BW as well as iops-max, but this requires you to know in advance exactly what workload is running in the group. The moment workoload changes, these settings might have a very bad effects.
So my biggest concern with max-bwidth and max-iops limits is that how will one configure the system for a dynamic environment. Think of two virtual machines being used by two customers. At one point they might be doing some copy operation and running sequential workload an later some webserver or database query might be doing some random read operations.
- Notice the interesting case of 16 random readers. iops for random reader group is really low, but still the throughput and iops of sequential reader group is very bad. I suspect that at CFQ level, some kind of mixup has taken place where we have not enabled idling for sequential reader and disk became seek bound hence both the group are loosing. (Just a guess)
Out of curiousity I looked at the results of set1 and set3 also and they seem to be exhibiting the similar behavior.
Set1
----
io-throttle + CFQ
-----------------------------------
BW limit group1=10 MB/s BW limit group2=10 MB/s [Multiple Random Reader] [Sequential Reader] nr Agg-bandw Max-latency Agg-iops nr Agg-bandw Max-latency Agg-iops 1 37KB/s 227K usec 9 1 8033KB/s 18773 usec 2008 2 342KB/s 601K usec 84 1 7406KB/s 476K usec 1851 4 677KB/s 163K usec 167 1 6743KB/s 69196 usec 1685 8 310KB/s 1780 msec 74 1 882KB/s 915K usec 220 16 877KB/s 431K usec 211 1 3278KB/s 274K usec 819 32 1109KB/s 1823 msec 261 1 1217KB/s 1022K usec 304
Set3
----
io-throttle + CFQ
-----------------------------------
BW limit group1=10 MB/s BW limit group2=10 MB/s [Multiple Random Reader] [Sequential Reader] nr Agg-bandw Max-latency Agg-iops nr Agg-bandw Max-latency Agg-iops 1 34KB/s 693K usec 8 1 7908KB/s 469K usec 1977 2 343KB/s 204K usec 85 1 7402KB/s 33962 usec 1850 4 691KB/s 228K usec 171 1 6847KB/s 76957 usec 1711 8 306KB/s 1806 msec 73 1 852KB/s 925K usec 213 16 287KB/s 3581 msec 63 1 439KB/s 1820K usec 109 32 976KB/s 3592K usec 230 1 1170KB/s 2895K usec 292
Quoted 65 lines
Quoted 57 lines
Multiple Sequential Reader vs Random Reader
===============================================
Now running a reverse test where in one group I am running increasing number of sequential readers and in other group I am running one random reader and see the impact of sequential readers on random reader.Vanilla CFQ
-----------------------------------
[Multiple Sequential Reader] [Random Reader]nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 13978KB/s 13978KB/s 13650KB/s 27614 usec 1 22KB/s 227 msec 2 6225KB/s 6166KB/s 12101KB/s 568K usec 1 10KB/s 457 msec 4 4052KB/s 2462KB/s 13107KB/s 322K usec 1 6KB/s 841 msec 8 1899KB/s 557KB/s 12960KB/s 829K usec 1 13KB/s 1628 msec 16 1007KB/s 279KB/s 13833KB/s 1629K usec 1 10KB/s 3236 msec 32 506KB/s 98KB/s 13704KB/s 3389K usec 1 6KB/s 3238 msecIO scheduler controller + CFQ
-----------------------------------
[Multiple Sequential Reader] [Random Reader]nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 5721KB/s 5721KB/s 5587KB/s 126K usec 1 223KB/s 126K usec 2 3216KB/s 1442KB/s 4549KB/s 349K usec 1 224KB/s 176K usec 4 1895KB/s 640KB/s 5121KB/s 775K usec 1 222KB/s 189K usec 8 957KB/s 285KB/s 6368KB/s 1680K usec 1 223KB/s 142K usec 16 458KB/s 132KB/s 6455KB/s 3343K usec 1 219KB/s 165K usec 32 248KB/s 55KB/s 6001KB/s 6957K usec 1 220KB/s 504K usecNotes:
- Random reader is well isolated from increasing number of sequential readers in other group. BW and latencies are stable.io-throttle + CFQ
-----------------------------------
BW limit group1=10 MB/s BW limit group2=10 MB/s
[Multiple Sequential Reader] [Random Reader]nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 8200KB/s 8200KB/s 8007KB/s 20275 usec 1 37KB/s 217K usec 2 3926KB/s 3919KB/s 7661KB/s 122K usec 1 16KB/s 441 msec 4 2271KB/s 1497KB/s 7672KB/s 611K usec 1 9KB/s 927 msec 8 1113KB/s 513KB/s 7507KB/s 849K usec 1 21KB/s 1020 msec 16 661KB/s 236KB/s 7959KB/s 1679K usec 1 13KB/s 2926 msec 32 292KB/s 109KB/s 7864KB/s 3446K usec 1 8KB/s 3439 msecBW limit group1=5 MB/s BW limit group2=5 MB/s
[Multiple Sequential Reader] [Random Reader]nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 4686KB/s 4686KB/s 4576KB/s 21095 usec 1 57KB/s 219K usec 2 2298KB/s 2179KB/s 4372KB/s 132K usec 1 37KB/s 431K usec 4 1245KB/s 1019KB/s 4449KB/s 324K usec 1 26KB/s 835 msec 8 584KB/s 403KB/s 4109KB/s 833K usec 1 30KB/s 1625K usec 16 346KB/s 252KB/s 4605KB/s 1641K usec 1 129KB/s 3236K usec 32 175KB/s 56KB/s 4269KB/s 3236K usec 1 8KB/s 3235 msecNotes:
Quoted 7 lines
- Above result is surprising to me. I have run it twice. In first run, I setup per cgroup limit as 10MB/s and in second run I set it up 5MB/s. In both the cases as number of sequential readers increase in other groups, random reader's throughput decreases and latencies increase. This is happening despite the fact that sequential readers are being throttled to make sure it does not impact workload in other group. Wondering why random readers are not seeing consistent throughput and latencies.
Quoted 5 lines
Maybe because CFQ is still trying to be fair among processes instead of cgroups. Remember that io-throttle doesn't touch the CFQ code (for this I'm definitely convinced that CFQ should be changed to think also in terms of cgroups, and io-throttle alone is not enough).
True. I think that's what is happening here. CFQ will see requests from all the sequential readers and will try to give these 100ms slice but random reader will get one chance to dispatch requests and then will again be at the back of the service tree.
Throttling at higher layers should help a bit so that group1 does not get to run for too long, but still it does not seem to be helping a lot.
So it becomes important that underying IO scheduler knows about groups and then does the scheduling accordingly otherwise we run into issues of "weak isolation" between groups and "not improved latecies".
Quoted 74 lines
So, even if group1 is being throttled in part it is still able to submit some requests that get a higher priority respect to the requests submitted by the single random reader task.
It could be interesting to test another IO scheduler (deadline, as or even noop) to check if this is the actual problem.
Quoted 6 lines
- Andrea, can you please also run similar tests to see if you see same results or not. This is to rule out any testing methodology errors or scripting bugs. :-). I also have collected the snapshot of some cgroup files like bandwidth-max, throttlecnt, and stats. Let me know if you want those to see what is happenig here.
Sure, I'll do some tests ASAP. Another interesting test would be to set a blockio.iops-max limit also for the sequential readers' cgroup, to be sure we're not touching some iops physical disk limit.
Could you post all the options you used with fio, so I can repeat some tests as similar as possible to yours?
Quoted 45 lines
Multiple Sequential Reader vs Sequential Reader
===============================================
- This time running random readers are out of the picture and trying to see the effect of increasing number of sequential readers on another sequential reader running in a different group.Vanilla CFQ
-----------------------------------
[Multiple Sequential Reader] [Sequential Reader]nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 6325KB/s 6325KB/s 6176KB/s 114K usec 1 6902KB/s 120K usec 2 4588KB/s 3102KB/s 7510KB/s 571K usec 1 4564KB/s 680K usec 4 3242KB/s 1158KB/s 9469KB/s 495K usec 1 3198KB/s 410K usec 8 1775KB/s 459KB/s 12011KB/s 1178K usec 1 1366KB/s 818K usec 16 943KB/s 296KB/s 13285KB/s 1923K usec 1 728KB/s 1816K usec 32 511KB/s 148KB/s 13555KB/s 3286K usec 1 391KB/s 3212K usecIO scheduler controller + CFQ
-----------------------------------
[Multiple Sequential Reader] [Sequential Reader]nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 6781KB/s 6781KB/s 6622KB/s 109K usec 1 6691KB/s 115K usec 2 3758KB/s 1876KB/s 5502KB/s 693K usec 1 6373KB/s 419K usec 4 2100KB/s 671KB/s 5751KB/s 987K usec 1 6330KB/s 569K usec 8 1023KB/s 355KB/s 6969KB/s 1569K usec 1 6086KB/s 120K usec 16 520KB/s 130KB/s 7094KB/s 3140K usec 1 5984KB/s 119K usec 32 245KB/s 86KB/s 6621KB/s 6571K usec 1 5850KB/s 113K usecNotes:
- BW and latencies of sequential reader in group 2 are fairly stable as number of readers increase in first group.io-throttle + CFQ
-----------------------------------
BW limit group1=30 MB/s BW limit group2=30 MB/s
[Multiple Sequential Reader] [Sequential Reader]nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 6343KB/s 6343KB/s 6195KB/s 116K usec 1 6993KB/s 109K usec 2 4583KB/s 3046KB/s 7451KB/s 583K usec 1 4516KB/s 433K usec 4 2945KB/s 1324KB/s 9552KB/s 602K usec 1 3001KB/s 583K usec 8 1804KB/s 473KB/s 12257KB/s 861K usec 1 1386KB/s 815K usec 16 942KB/s 265KB/s 13560KB/s 1659K usec 1 718KB/s 1658K usec 32 462KB/s 143KB/s 13757KB/s 3482K usec 1 409KB/s 3480K usecQuoted 7 lines
Notes:
- BW decreases and latencies increase in group2 as number of readers increase in first group. This should be due to fact that no throttling will happen as none of the groups is hitting the limit of 30MB/s. To me this is the tricky part. How a service provider is supposed to set the limit of groups. If groups are not hitting max limits, it will still impact the BW and latencies in other group.
Quoted 4 lines
Are you using 4k block size here? because in case of too small blocks you could hit some physical iops limit. Also for this case it could be interesting to see what happens setting both BW and iops hard limits.
Hmm.., Same results posted with iops numbers.
io-throttle + CFQ
-----------------------------------
BW limit group1=30 MB/s BW limit group2=30 MB/s [Multiple Sequential Reader] [Sequential Reader] nr Agg-bandw Max-latency Agg-iops nr Agg-bandw Max-latency Agg-iops 1 6195KB/s 116K usec 1548 1 6993KB/s 109K usec 1748 2 7451KB/s 583K usec 1862 1 4516KB/s 433K usec 1129 4 9552KB/s 602K usec 2387 1 3001KB/s 583K usec 750 8 12257KB/s 861K usec 3060 1 1386KB/s 815K usec 346 16 13560KB/s 1659K usec 3382 1 718KB/s 1658K usec 179 32 13757KB/s 3482K usec 3422 1 409KB/s 3480K usec 102
BW limit group1=10 MB/s BW limit group2=10 MB/s [Multiple Sequential Reader] [Sequential Reader] nr Agg-bandw Max-latency Agg-iops nr Agg-bandw Max-latency Agg-iops 1 4032KB/s 215K usec 1008 1 4076KB/s 170K usec 1019 2 4655KB/s 291K usec 1163 1 2891KB/s 212K usec 722 4 5872KB/s 417K usec 1466 1 1881KB/s 411K usec 470 8 7312KB/s 841K usec 1824 1 853KB/s 816K usec 213 16 7844KB/s 1728K usec 1956 1 503KB/s 1609K usec 125 32 7920KB/s 3417K usec 1969 1 249KB/s 3205K usec 62
BW limit group1=5 MB/s BW limit group2=5 MB/s [Multiple Sequential Reader] [Sequential Reader] nr Agg-bandw Max-latency Agg-iops nr Agg-bandw Max-latency Agg-iops 1 2377KB/s 110K usec 594 1 2415KB/s 120K usec 603 2 2759KB/s 222K usec 689 1 1709KB/s 220K usec 427 4 3314KB/s 420K usec 828 1 1163KB/s 414K usec 290 8 4060KB/s 901K usec 1011 1 527KB/s 816K usec 131 16 4324KB/s 1613K usec 1074 1 311KB/s 1613K usec 77 32 4320KB/s 3235K usec 1067 1 163KB/s 3209K usec 40
Note that with bw limit 30MB/s, we are able to hit iops more than 3400 but with bw=5MB/s, we are hitting close to 1100 iops. So I think we are under-utilizing the storage here and not run into any kind of iops limit.
Quoted 11 lines
Quoted 11 lines
BW limit group1=10 MB/s BW limit group2=10 MB/s
[Multiple Sequential Reader] [Sequential Reader]nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 4128KB/s 4128KB/s 4032KB/s 215K usec 1 4076KB/s 170K usec 2 2880KB/s 1886KB/s 4655KB/s 291K usec 1 2891KB/s 212K usec 4 1912KB/s 888KB/s 5872KB/s 417K usec 1 1881KB/s 411K usec 8 1032KB/s 432KB/s 7312KB/s 841K usec 1 853KB/s 816K usec 16 540KB/s 259KB/s 7844KB/s 1728K usec 1 503KB/s 1609K usec 32 291KB/s 111KB/s 7920KB/s 3417K usec 1 249KB/s 3205K usec
Quoted 8 lines
Quoted 4 lines
Notes:
- Same test with 10MB/s as group limit. This is again a surprising result. Max BW in first group is being throttled but still throughput is dropping significantly in second group and latencies are on the rise.Same consideration about CFQ and/or iops limit. Could you post all the fio options you've used also for this test (or better, for all tests)?
Already posted in a separate mail.
Quoted 6 lines
Quoted 3 lines
- Limit of first group is 10MB/s but it is achieving max BW of around 8MB/s only. What happened to rest of the 2MB/s?
Ditto.
For 10MB/s case, max iops seems to be 2000 collectively, way below than 3400. So I doubt that this is case of hitting max iops.
Thanks
Vivek
Re: Performance numbers with IO throttling patches (Was: Re: IO scheduler based IO controller V10)
On Mon, Oct 12, 2009 at 05:11:20PM -0400, Vivek Goyal wrote:
[snip]
Quoted 42 lines
I modified my report scripts to also output aggreagate iops numbers and remove max-bandwidth and min-bandwidth numbers. So for same tests and same results I am now reporting iops numbers also. ( I have not re-run the tests.)
IO scheduler controller + CFQ
-----------------------------------[Multiple Random Reader] [Sequential Reader] nr Agg-bandw Max-latency Agg-iops nr Agg-bandw Max-latency Agg-iops 1 223KB/s 132K usec 55 1 5551KB/s 129K usec 1387 2 190KB/s 154K usec 46 1 5718KB/s 122K usec 1429 4 445KB/s 208K usec 111 1 5909KB/s 116K usec 1477 8 158KB/s 2820 msec 36 1 5445KB/s 168K usec 1361 16 145KB/s 5963 msec 28 1 5418KB/s 164K usec 1354 32 139KB/s 12762 msec 23 1 5398KB/s 175K usec 1349io-throttle + CFQ
-----------------------------------BW limit group1=10 MB/s BW limit group2=10 MB/s [Multiple Random Reader] [Sequential Reader] nr Agg-bandw Max-latency Agg-iops nr Agg-bandw Max-latency Agg-iops 1 36KB/s 218K usec 9 1 8006KB/s 20529 usec 2001 2 360KB/s 228K usec 89 1 7475KB/s 33665 usec 1868 4 699KB/s 262K usec 173 1 6800KB/s 46224 usec 1700 8 573KB/s 1800K usec 139 1 2835KB/s 885K usec 708 16 294KB/s 3590 msec 68 1 437KB/s 1855K usec 109 32 980KB/s 2861K usec 230 1 1145KB/s 1952K usec 286Note that in case of random reader groups, iops are really small. Few thougts.
- What should be the iops limit I should choose for the group. Lets say if I choose "80", then things should be better for sequential reader group, but just think of what will happen to random reader group. Especially, if nature of workload in group1 changes to sequential. Group1 will simply be killed.
So yes, one can limit a group both by BW as well as iops-max, but this requires you to know in advance exactly what workload is running in the group. The moment workoload changes, these settings might have a very bad effects.
Quoted 5 lines
So my biggest concern with max-bwidth and max-iops limits is that how will one configure the system for a dynamic environment. Think of two virtual machines being used by two customers. At one point they might be doing some copy operation and running sequential workload an later some webserver or database query might be doing some random read operations.
The main problem IMHO is how to accurately evaluate the cost of an IO operation. On rotational media for example the cost to read two distant blocks is not the same cost of reading two contiguous blocks (while on a flash/SSD drive the cost is probably the same).
io-throttle tries to quantify the cost in absolute terms (iops and BW), but this is not enough to cover all the possible cases. For example, you could hit a physical disk limit, because you're doing a workload too seeky, even if the iops and BW numbers are low.
Quoted 7 lines
- Notice the interesting case of 16 random readers. iops for random reader group is really low, but still the throughput and iops of sequential reader group is very bad. I suspect that at CFQ level, some kind of mixup has taken place where we have not enabled idling for sequential reader and disk became seek bound hence both the group are loosing. (Just a guess)
Yes, my guess is the same.
I've re-run some of your tests using a SSD (a MOBI MTRON MSD-PATA3018-ZIF1), but changing few parameters: I used a larger block size for the sequential workload (there's no need to reduce the block size of the single reads if we suppose to read a lot of contiguous blocks).
And for all the io-throttle tests I switched to noop scheduler (CFQ must be changed to be cgroup-aware before using it together with io-throttle, otherwise the result is that one simply breaks the logic of the other).
=== io-throttle settings ===
cgroup #1: max-bw 10MB/s, max-iops 2150 iop/s
cgroup #1: max-bw 10MB/s, max-iops 2150 iop/s
During the tests I used a larger block size for sequential readers, respect to the random readers:
sequential-read: block size = 1MB
random-read: block size = 4KB
sequential-readers vs sequential-reader
=======================================
[ cgroup #1 workload ]
fio_args="--rw=read --bs=1M --size=512M --runtime=30 --numjobs=N --direct=1" [ cgroup #2 workload ]
fio_args="--rw=read --bs=1M --size=512M --runtime=30 --numjobs=1 --direct=1"
__2.6.32-rc5__
[ cgroup #1 ] [ cgroup #2 ]
tasks aggr-bw tasks aggr-bw
1 36210KB/s 1 36992KB/s
2 47558KB/s 1 24479KB/s
4 57587KB/s 1 14809KB/s
8 64667KB/s 1 8393KB/s
__2.6.32-rc5-io-throttle__
[ cgroup #1 ] [ cgroup #2 ]
tasks aggr-bw tasks aggr-bw
1 10195KB/s 1 10193KB/s
2 10279KB/s 1 10276KB/s
4 10281KB/s 1 10277KB/s
8 10279KB/s 1 10277KB/s
random-readers vs sequential-reader
===================================
[ cgroup #1 workload ]
fio_args="--rw=randread --bs=4k --size=512M --runtime=30 --numjobs=N --direct=1" [ cgroup #2 workload ]
fio_args="--rw=read --bs=1M --size=512M --runtime=30 --numjobs=1 --direct=1"
__2.6.32-rc5__
[ cgroup #1 ] [ cgroup #2 ]
tasks aggr-bw tasks aggr-bw
1 4767KB/s 1 52819KB/s
2 5900KB/s 1 39788KB/s
4 7783KB/s 1 27966KB/s
8 9296KB/s 1 17606KB/s
__2.6.32-rc5-io-throttle__
[ cgroup #1 ] [ cgroup #2 ]
tasks aggr-bw tasks aggr-bw
1 8861KB/s 1 8886KB/s
2 8887KB/s 1 7578KB/s
4 8886KB/s 1 7271KB/s
8 8889KB/s 1 7489KB/s
sequential-readers vs random-reader
===================================
[ cgroup #1 workload ]
fio_args="--rw=read --bs=1M --size=512M --runtime=30 --numjobs=N --direct=1" [ cgroup #2 workload ]
fio_args="--rw=randread --bs=4k --size=512M --runtime=30 --numjobs=1 --direct=1"
__2.6.32-rc5__
[ cgroup #1 ] [ cgroup #2 ]
tasks aggr-bw tasks aggr-bw
1 54511KB/s 1 4865KB/s
2 70312KB/s 1 965KB/s
4 71543KB/s 1 484KB/s
8 72899KB/s 1 98KB/s
__2.6.32-rc5-io-throttle__
[ cgroup #1 ] [ cgroup #2 ]
tasks aggr-bw tasks aggr-bw
1 8875KB/s 1 8885KB/s
2 8884KB/s 1 8148KB/s
4 8886KB/s 1 7637KB/s
8 8886KB/s 1 7411KB/s
random-readers vs random-reader
===============================
[ cgroup #1 workload ]
fio_args="--rw=randread --bs=4k --size=512M --runtime=30 --numjobs=N --direct=1" [ cgroup #2 workload ]
fio_args="--rw=randread --bs=4k --size=512M --runtime=30 --numjobs=1 --direct=1"
__2.6.32-rc5__
[ cgroup #1 ] [ cgroup #2 ]
tasks aggr-bw tasks aggr-bw
1 6141KB/s 1 6320KB/s
2 8567KB/s 1 3987KB/s
4 9783KB/s 1 2610KB/s
8 11067KB/s 1 1227KB/s
__2.6.32-rc5-io-throttle__
[ cgroup #1 ] [ cgroup #2 ]
tasks aggr-bw tasks aggr-bw
1 8883KB/s 1 8886KB/s
2 8888KB/s 1 7676KB/s
4 8887KB/s 1 7364KB/s
8 8884KB/s 1 7264KB/s
With the SSD there's not a consistent degradation of cgroup #2 when we increase the tasks of the concurrent random readers in cgroup #1 (both in the random-vs-random or random-vs-sequential cases).
We should better analyze the details (probably blktrace would help here), but it seems that in your tests the mix of CFQ and io-throttle generated a too seeky workload that caused the bad performance values of the sequential reader.
-Andrea
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: Performance numbers with IO throttling patches (Was: Re: IO scheduler based IO controller V10)
On Mon, Oct 12, 2009 at 05:11:20PM -0400, Vivek Goyal wrote:
[snip]
Quoted 42 lines
I modified my report scripts to also output aggreagate iops numbers and remove max-bandwidth and min-bandwidth numbers. So for same tests and same results I am now reporting iops numbers also. ( I have not re-run the tests.)
IO scheduler controller + CFQ
-----------------------------------[Multiple Random Reader] [Sequential Reader] nr Agg-bandw Max-latency Agg-iops nr Agg-bandw Max-latency Agg-iops 1 223KB/s 132K usec 55 1 5551KB/s 129K usec 1387 2 190KB/s 154K usec 46 1 5718KB/s 122K usec 1429 4 445KB/s 208K usec 111 1 5909KB/s 116K usec 1477 8 158KB/s 2820 msec 36 1 5445KB/s 168K usec 1361 16 145KB/s 5963 msec 28 1 5418KB/s 164K usec 1354 32 139KB/s 12762 msec 23 1 5398KB/s 175K usec 1349io-throttle + CFQ
-----------------------------------BW limit group1=10 MB/s BW limit group2=10 MB/s [Multiple Random Reader] [Sequential Reader] nr Agg-bandw Max-latency Agg-iops nr Agg-bandw Max-latency Agg-iops 1 36KB/s 218K usec 9 1 8006KB/s 20529 usec 2001 2 360KB/s 228K usec 89 1 7475KB/s 33665 usec 1868 4 699KB/s 262K usec 173 1 6800KB/s 46224 usec 1700 8 573KB/s 1800K usec 139 1 2835KB/s 885K usec 708 16 294KB/s 3590 msec 68 1 437KB/s 1855K usec 109 32 980KB/s 2861K usec 230 1 1145KB/s 1952K usec 286Note that in case of random reader groups, iops are really small. Few thougts.
- What should be the iops limit I should choose for the group. Lets say if I choose "80", then things should be better for sequential reader group, but just think of what will happen to random reader group. Especially, if nature of workload in group1 changes to sequential. Group1 will simply be killed.
So yes, one can limit a group both by BW as well as iops-max, but this requires you to know in advance exactly what workload is running in the group. The moment workoload changes, these settings might have a very bad effects.
Quoted 5 lines
So my biggest concern with max-bwidth and max-iops limits is that how will one configure the system for a dynamic environment. Think of two virtual machines being used by two customers. At one point they might be doing some copy operation and running sequential workload an later some webserver or database query might be doing some random read operations.
The main problem IMHO is how to accurately evaluate the cost of an IO operation. On rotational media for example the cost to read two distant blocks is not the same cost of reading two contiguous blocks (while on a flash/SSD drive the cost is probably the same).
io-throttle tries to quantify the cost in absolute terms (iops and BW), but this is not enough to cover all the possible cases. For example, you could hit a physical disk limit, because you're doing a workload too seeky, even if the iops and BW numbers are low.
Quoted 7 lines
- Notice the interesting case of 16 random readers. iops for random reader group is really low, but still the throughput and iops of sequential reader group is very bad. I suspect that at CFQ level, some kind of mixup has taken place where we have not enabled idling for sequential reader and disk became seek bound hence both the group are loosing. (Just a guess)
Yes, my guess is the same.
I've re-run some of your tests using a SSD (a MOBI MTRON MSD-PATA3018-ZIF1), but changing few parameters: I used a larger block size for the sequential workload (there's no need to reduce the block size of the single reads if we suppose to read a lot of contiguous blocks).
And for all the io-throttle tests I switched to noop scheduler (CFQ must be changed to be cgroup-aware before using it together with io-throttle, otherwise the result is that one simply breaks the logic of the other).
=== io-throttle settings ===
cgroup #1: max-bw 10MB/s, max-iops 2150 iop/s
cgroup #1: max-bw 10MB/s, max-iops 2150 iop/s
During the tests I used a larger block size for sequential readers, respect to the random readers:
sequential-read: block size = 1MB
random-read: block size = 4KB
sequential-readers vs sequential-reader
=======================================
[ cgroup #1 workload ]
fio_args="--rw=read --bs=1M --size=512M --runtime=30 --numjobs=N --direct=1" [ cgroup #2 workload ]
fio_args="--rw=read --bs=1M --size=512M --runtime=30 --numjobs=1 --direct=1"
__2.6.32-rc5__
[ cgroup #1 ] [ cgroup #2 ]
tasks aggr-bw tasks aggr-bw
1 36210KB/s 1 36992KB/s
2 47558KB/s 1 24479KB/s
4 57587KB/s 1 14809KB/s
8 64667KB/s 1 8393KB/s
__2.6.32-rc5-io-throttle__
[ cgroup #1 ] [ cgroup #2 ]
tasks aggr-bw tasks aggr-bw
1 10195KB/s 1 10193KB/s
2 10279KB/s 1 10276KB/s
4 10281KB/s 1 10277KB/s
8 10279KB/s 1 10277KB/s
random-readers vs sequential-reader
===================================
[ cgroup #1 workload ]
fio_args="--rw=randread --bs=4k --size=512M --runtime=30 --numjobs=N --direct=1" [ cgroup #2 workload ]
fio_args="--rw=read --bs=1M --size=512M --runtime=30 --numjobs=1 --direct=1"
__2.6.32-rc5__
[ cgroup #1 ] [ cgroup #2 ]
tasks aggr-bw tasks aggr-bw
1 4767KB/s 1 52819KB/s
2 5900KB/s 1 39788KB/s
4 7783KB/s 1 27966KB/s
8 9296KB/s 1 17606KB/s
__2.6.32-rc5-io-throttle__
[ cgroup #1 ] [ cgroup #2 ]
tasks aggr-bw tasks aggr-bw
1 8861KB/s 1 8886KB/s
2 8887KB/s 1 7578KB/s
4 8886KB/s 1 7271KB/s
8 8889KB/s 1 7489KB/s
sequential-readers vs random-reader
===================================
[ cgroup #1 workload ]
fio_args="--rw=read --bs=1M --size=512M --runtime=30 --numjobs=N --direct=1" [ cgroup #2 workload ]
fio_args="--rw=randread --bs=4k --size=512M --runtime=30 --numjobs=1 --direct=1"
__2.6.32-rc5__
[ cgroup #1 ] [ cgroup #2 ]
tasks aggr-bw tasks aggr-bw
1 54511KB/s 1 4865KB/s
2 70312KB/s 1 965KB/s
4 71543KB/s 1 484KB/s
8 72899KB/s 1 98KB/s
__2.6.32-rc5-io-throttle__
[ cgroup #1 ] [ cgroup #2 ]
tasks aggr-bw tasks aggr-bw
1 8875KB/s 1 8885KB/s
2 8884KB/s 1 8148KB/s
4 8886KB/s 1 7637KB/s
8 8886KB/s 1 7411KB/s
random-readers vs random-reader
===============================
[ cgroup #1 workload ]
fio_args="--rw=randread --bs=4k --size=512M --runtime=30 --numjobs=N --direct=1" [ cgroup #2 workload ]
fio_args="--rw=randread --bs=4k --size=512M --runtime=30 --numjobs=1 --direct=1"
__2.6.32-rc5__
[ cgroup #1 ] [ cgroup #2 ]
tasks aggr-bw tasks aggr-bw
1 6141KB/s 1 6320KB/s
2 8567KB/s 1 3987KB/s
4 9783KB/s 1 2610KB/s
8 11067KB/s 1 1227KB/s
__2.6.32-rc5-io-throttle__
[ cgroup #1 ] [ cgroup #2 ]
tasks aggr-bw tasks aggr-bw
1 8883KB/s 1 8886KB/s
2 8888KB/s 1 7676KB/s
4 8887KB/s 1 7364KB/s
8 8884KB/s 1 7264KB/s
With the SSD there's not a consistent degradation of cgroup #2 when we increase the tasks of the concurrent random readers in cgroup #1 (both in the random-vs-random or random-vs-sequential cases).
We should better analyze the details (probably blktrace would help here), but it seems that in your tests the mix of CFQ and io-throttle generated a too seeky workload that caused the bad performance values of the sequential reader.
-Andrea
Re: Performance numbers with IO throttling patches (Was: Re: IO scheduler based IO controller V10)
In reply to Andrea RighiOn Sun, Oct 11, 2009 at 12:27:30AM +0200, Andrea Righi wrote:
[..]
Quoted 61 lines
Quoted 57 lines
Multiple Random Reader vs Sequential Reader
===============================================
Generally random readers bring the throughput down of others in the system. Ran a test to see the impact of increasing number of random readers on single sequential reader in different groups.Vanilla CFQ
-----------------------------------
[Multiple Random Reader] [Sequential Reader]nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 23KB/s 23KB/s 22KB/s 691 msec 1 13519KB/s 468K usec 2 152KB/s 152KB/s 297KB/s 244K usec 1 12380KB/s 31675 usec 4 174KB/s 156KB/s 638KB/s 249K usec 1 10860KB/s 36715 usec 8 49KB/s 11KB/s 310KB/s 1856 msec 1 1292KB/s 990K usec 16 63KB/s 48KB/s 877KB/s 762K usec 1 3905KB/s 506K usec 32 35KB/s 27KB/s 951KB/s 2655 msec 1 1109KB/s 1910K usecIO scheduler controller + CFQ
-----------------------------------
[Multiple Random Reader] [Sequential Reader]nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 228KB/s 228KB/s 223KB/s 132K usec 1 5551KB/s 129K usec 2 97KB/s 97KB/s 190KB/s 154K usec 1 5718KB/s 122K usec 4 115KB/s 110KB/s 445KB/s 208K usec 1 5909KB/s 116K usec 8 23KB/s 12KB/s 158KB/s 2820 msec 1 5445KB/s 168K usec 16 11KB/s 3KB/s 145KB/s 5963 msec 1 5418KB/s 164K usec 32 6KB/s 2KB/s 139KB/s 12762 msec 1 5398KB/s 175K usecNotes:
- Sequential reader in group2 seems to be well isolated from random readers in group1. Throughput and latency of sequential reader are stable and don't drop as number of random readers inrease in system.io-throttle + CFQ
------------------
BW limit group1=10 MB/s BW limit group2=10 MB/s
[Multiple Random Reader] [Sequential Reader]nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 37KB/s 37KB/s 36KB/s 218K usec 1 8006KB/s 20529 usec 2 185KB/s 183KB/s 360KB/s 228K usec 1 7475KB/s 33665 usec 4 188KB/s 171KB/s 699KB/s 262K usec 1 6800KB/s 46224 usec 8 84KB/s 51KB/s 573KB/s 1800K usec 1 2835KB/s 885K usec 16 21KB/s 9KB/s 294KB/s 3590 msec 1 437KB/s 1855K usec 32 34KB/s 27KB/s 980KB/s 2861K usec 1 1145KB/s 1952K usecNotes:
- I have setup limits of 10MB/s in both the cgroups. Now random reader group will never achieve that kind of speed, so it will not be throttled and then it goes onto impact the throughput and latency of other groups in the system.- Now the key question is how conservative one should in be setting up max BW limit. On this box if a customer has bought 10MB/s cgroup and if he is running some random readers it will kill throughput of other groups in the system and their latencies will shoot up. No isolation in this case.
Quoted 3 lines
- So in general, max BW provides isolation from high speed groups but it does not provide isolaton from random reader groups which are moving slow.
Quoted 6 lines
Remember that in addition to blockio.bandwidth-max the io-throttle controlller also provides blockio.iops-max to enforce hard limits on the number of IO operations per second. Probably for this testcase both cgroups should be limited in terms of BW and iops to achieve a better isolation.
I modified my report scripts to also output aggreagate iops numbers and remove max-bandwidth and min-bandwidth numbers. So for same tests and same results I am now reporting iops numbers also. ( I have not re-run the tests.)
IO scheduler controller + CFQ
-----------------------------------
[Multiple Random Reader] [Sequential Reader] nr Agg-bandw Max-latency Agg-iops nr Agg-bandw Max-latency Agg-iops 1 223KB/s 132K usec 55 1 5551KB/s 129K usec 1387 2 190KB/s 154K usec 46 1 5718KB/s 122K usec 1429 4 445KB/s 208K usec 111 1 5909KB/s 116K usec 1477 8 158KB/s 2820 msec 36 1 5445KB/s 168K usec 1361 16 145KB/s 5963 msec 28 1 5418KB/s 164K usec 1354 32 139KB/s 12762 msec 23 1 5398KB/s 175K usec 1349
io-throttle + CFQ
-----------------------------------
BW limit group1=10 MB/s BW limit group2=10 MB/s [Multiple Random Reader] [Sequential Reader] nr Agg-bandw Max-latency Agg-iops nr Agg-bandw Max-latency Agg-iops 1 36KB/s 218K usec 9 1 8006KB/s 20529 usec 2001 2 360KB/s 228K usec 89 1 7475KB/s 33665 usec 1868 4 699KB/s 262K usec 173 1 6800KB/s 46224 usec 1700 8 573KB/s 1800K usec 139 1 2835KB/s 885K usec 708 16 294KB/s 3590 msec 68 1 437KB/s 1855K usec 109 32 980KB/s 2861K usec 230 1 1145KB/s 1952K usec 286
Note that in case of random reader groups, iops are really small. Few thougts.
- What should be the iops limit I should choose for the group. Lets say if I choose "80", then things should be better for sequential reader group, but just think of what will happen to random reader group. Especially, if nature of workload in group1 changes to sequential. Group1 will simply be killed.
So yes, one can limit a group both by BW as well as iops-max, but this requires you to know in advance exactly what workload is running in the group. The moment workoload changes, these settings might have a very bad effects.
So my biggest concern with max-bwidth and max-iops limits is that how will one configure the system for a dynamic environment. Think of two virtual machines being used by two customers. At one point they might be doing some copy operation and running sequential workload an later some webserver or database query might be doing some random read operations.
- Notice the interesting case of 16 random readers. iops for random reader group is really low, but still the throughput and iops of sequential reader group is very bad. I suspect that at CFQ level, some kind of mixup has taken place where we have not enabled idling for sequential reader and disk became seek bound hence both the group are loosing. (Just a guess)
Out of curiousity I looked at the results of set1 and set3 also and they seem to be exhibiting the similar behavior.
Set1
----
io-throttle + CFQ
-----------------------------------
BW limit group1=10 MB/s BW limit group2=10 MB/s [Multiple Random Reader] [Sequential Reader] nr Agg-bandw Max-latency Agg-iops nr Agg-bandw Max-latency Agg-iops 1 37KB/s 227K usec 9 1 8033KB/s 18773 usec 2008 2 342KB/s 601K usec 84 1 7406KB/s 476K usec 1851 4 677KB/s 163K usec 167 1 6743KB/s 69196 usec 1685 8 310KB/s 1780 msec 74 1 882KB/s 915K usec 220 16 877KB/s 431K usec 211 1 3278KB/s 274K usec 819 32 1109KB/s 1823 msec 261 1 1217KB/s 1022K usec 304
Set3
----
io-throttle + CFQ
-----------------------------------
BW limit group1=10 MB/s BW limit group2=10 MB/s [Multiple Random Reader] [Sequential Reader] nr Agg-bandw Max-latency Agg-iops nr Agg-bandw Max-latency Agg-iops 1 34KB/s 693K usec 8 1 7908KB/s 469K usec 1977 2 343KB/s 204K usec 85 1 7402KB/s 33962 usec 1850 4 691KB/s 228K usec 171 1 6847KB/s 76957 usec 1711 8 306KB/s 1806 msec 73 1 852KB/s 925K usec 213 16 287KB/s 3581 msec 63 1 439KB/s 1820K usec 109 32 976KB/s 3592K usec 230 1 1170KB/s 2895K usec 292
Quoted 65 lines
Quoted 57 lines
Multiple Sequential Reader vs Random Reader
===============================================
Now running a reverse test where in one group I am running increasing number of sequential readers and in other group I am running one random reader and see the impact of sequential readers on random reader.Vanilla CFQ
-----------------------------------
[Multiple Sequential Reader] [Random Reader]nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 13978KB/s 13978KB/s 13650KB/s 27614 usec 1 22KB/s 227 msec 2 6225KB/s 6166KB/s 12101KB/s 568K usec 1 10KB/s 457 msec 4 4052KB/s 2462KB/s 13107KB/s 322K usec 1 6KB/s 841 msec 8 1899KB/s 557KB/s 12960KB/s 829K usec 1 13KB/s 1628 msec 16 1007KB/s 279KB/s 13833KB/s 1629K usec 1 10KB/s 3236 msec 32 506KB/s 98KB/s 13704KB/s 3389K usec 1 6KB/s 3238 msecIO scheduler controller + CFQ
-----------------------------------
[Multiple Sequential Reader] [Random Reader]nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 5721KB/s 5721KB/s 5587KB/s 126K usec 1 223KB/s 126K usec 2 3216KB/s 1442KB/s 4549KB/s 349K usec 1 224KB/s 176K usec 4 1895KB/s 640KB/s 5121KB/s 775K usec 1 222KB/s 189K usec 8 957KB/s 285KB/s 6368KB/s 1680K usec 1 223KB/s 142K usec 16 458KB/s 132KB/s 6455KB/s 3343K usec 1 219KB/s 165K usec 32 248KB/s 55KB/s 6001KB/s 6957K usec 1 220KB/s 504K usecNotes:
- Random reader is well isolated from increasing number of sequential readers in other group. BW and latencies are stable.io-throttle + CFQ
-----------------------------------
BW limit group1=10 MB/s BW limit group2=10 MB/s
[Multiple Sequential Reader] [Random Reader]nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 8200KB/s 8200KB/s 8007KB/s 20275 usec 1 37KB/s 217K usec 2 3926KB/s 3919KB/s 7661KB/s 122K usec 1 16KB/s 441 msec 4 2271KB/s 1497KB/s 7672KB/s 611K usec 1 9KB/s 927 msec 8 1113KB/s 513KB/s 7507KB/s 849K usec 1 21KB/s 1020 msec 16 661KB/s 236KB/s 7959KB/s 1679K usec 1 13KB/s 2926 msec 32 292KB/s 109KB/s 7864KB/s 3446K usec 1 8KB/s 3439 msecBW limit group1=5 MB/s BW limit group2=5 MB/s
[Multiple Sequential Reader] [Random Reader]nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 4686KB/s 4686KB/s 4576KB/s 21095 usec 1 57KB/s 219K usec 2 2298KB/s 2179KB/s 4372KB/s 132K usec 1 37KB/s 431K usec 4 1245KB/s 1019KB/s 4449KB/s 324K usec 1 26KB/s 835 msec 8 584KB/s 403KB/s 4109KB/s 833K usec 1 30KB/s 1625K usec 16 346KB/s 252KB/s 4605KB/s 1641K usec 1 129KB/s 3236K usec 32 175KB/s 56KB/s 4269KB/s 3236K usec 1 8KB/s 3235 msecNotes:
Quoted 7 lines
- Above result is surprising to me. I have run it twice. In first run, I setup per cgroup limit as 10MB/s and in second run I set it up 5MB/s. In both the cases as number of sequential readers increase in other groups, random reader's throughput decreases and latencies increase. This is happening despite the fact that sequential readers are being throttled to make sure it does not impact workload in other group. Wondering why random readers are not seeing consistent throughput and latencies.
Quoted 5 lines
Maybe because CFQ is still trying to be fair among processes instead of cgroups. Remember that io-throttle doesn't touch the CFQ code (for this I'm definitely convinced that CFQ should be changed to think also in terms of cgroups, and io-throttle alone is not enough).
True. I think that's what is happening here. CFQ will see requests from all the sequential readers and will try to give these 100ms slice but random reader will get one chance to dispatch requests and then will again be at the back of the service tree.
Throttling at higher layers should help a bit so that group1 does not get to run for too long, but still it does not seem to be helping a lot.
So it becomes important that underying IO scheduler knows about groups and then does the scheduling accordingly otherwise we run into issues of "weak isolation" between groups and "not improved latecies".
Quoted 74 lines
So, even if group1 is being throttled in part it is still able to submit some requests that get a higher priority respect to the requests submitted by the single random reader task.
It could be interesting to test another IO scheduler (deadline, as or even noop) to check if this is the actual problem.
Quoted 6 lines
- Andrea, can you please also run similar tests to see if you see same results or not. This is to rule out any testing methodology errors or scripting bugs. :-). I also have collected the snapshot of some cgroup files like bandwidth-max, throttlecnt, and stats. Let me know if you want those to see what is happenig here.
Sure, I'll do some tests ASAP. Another interesting test would be to set a blockio.iops-max limit also for the sequential readers' cgroup, to be sure we're not touching some iops physical disk limit.
Could you post all the options you used with fio, so I can repeat some tests as similar as possible to yours?
Quoted 45 lines
Multiple Sequential Reader vs Sequential Reader
===============================================
- This time running random readers are out of the picture and trying to see the effect of increasing number of sequential readers on another sequential reader running in a different group.Vanilla CFQ
-----------------------------------
[Multiple Sequential Reader] [Sequential Reader]nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 6325KB/s 6325KB/s 6176KB/s 114K usec 1 6902KB/s 120K usec 2 4588KB/s 3102KB/s 7510KB/s 571K usec 1 4564KB/s 680K usec 4 3242KB/s 1158KB/s 9469KB/s 495K usec 1 3198KB/s 410K usec 8 1775KB/s 459KB/s 12011KB/s 1178K usec 1 1366KB/s 818K usec 16 943KB/s 296KB/s 13285KB/s 1923K usec 1 728KB/s 1816K usec 32 511KB/s 148KB/s 13555KB/s 3286K usec 1 391KB/s 3212K usecIO scheduler controller + CFQ
-----------------------------------
[Multiple Sequential Reader] [Sequential Reader]nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 6781KB/s 6781KB/s 6622KB/s 109K usec 1 6691KB/s 115K usec 2 3758KB/s 1876KB/s 5502KB/s 693K usec 1 6373KB/s 419K usec 4 2100KB/s 671KB/s 5751KB/s 987K usec 1 6330KB/s 569K usec 8 1023KB/s 355KB/s 6969KB/s 1569K usec 1 6086KB/s 120K usec 16 520KB/s 130KB/s 7094KB/s 3140K usec 1 5984KB/s 119K usec 32 245KB/s 86KB/s 6621KB/s 6571K usec 1 5850KB/s 113K usecNotes:
- BW and latencies of sequential reader in group 2 are fairly stable as number of readers increase in first group.io-throttle + CFQ
-----------------------------------
BW limit group1=30 MB/s BW limit group2=30 MB/s
[Multiple Sequential Reader] [Sequential Reader]nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 6343KB/s 6343KB/s 6195KB/s 116K usec 1 6993KB/s 109K usec 2 4583KB/s 3046KB/s 7451KB/s 583K usec 1 4516KB/s 433K usec 4 2945KB/s 1324KB/s 9552KB/s 602K usec 1 3001KB/s 583K usec 8 1804KB/s 473KB/s 12257KB/s 861K usec 1 1386KB/s 815K usec 16 942KB/s 265KB/s 13560KB/s 1659K usec 1 718KB/s 1658K usec 32 462KB/s 143KB/s 13757KB/s 3482K usec 1 409KB/s 3480K usecQuoted 7 lines
Notes:
- BW decreases and latencies increase in group2 as number of readers increase in first group. This should be due to fact that no throttling will happen as none of the groups is hitting the limit of 30MB/s. To me this is the tricky part. How a service provider is supposed to set the limit of groups. If groups are not hitting max limits, it will still impact the BW and latencies in other group.
Quoted 4 lines
Are you using 4k block size here? because in case of too small blocks you could hit some physical iops limit. Also for this case it could be interesting to see what happens setting both BW and iops hard limits.
Hmm.., Same results posted with iops numbers.
io-throttle + CFQ
-----------------------------------
BW limit group1=30 MB/s BW limit group2=30 MB/s [Multiple Sequential Reader] [Sequential Reader] nr Agg-bandw Max-latency Agg-iops nr Agg-bandw Max-latency Agg-iops 1 6195KB/s 116K usec 1548 1 6993KB/s 109K usec 1748 2 7451KB/s 583K usec 1862 1 4516KB/s 433K usec 1129 4 9552KB/s 602K usec 2387 1 3001KB/s 583K usec 750 8 12257KB/s 861K usec 3060 1 1386KB/s 815K usec 346 16 13560KB/s 1659K usec 3382 1 718KB/s 1658K usec 179 32 13757KB/s 3482K usec 3422 1 409KB/s 3480K usec 102
BW limit group1=10 MB/s BW limit group2=10 MB/s [Multiple Sequential Reader] [Sequential Reader] nr Agg-bandw Max-latency Agg-iops nr Agg-bandw Max-latency Agg-iops 1 4032KB/s 215K usec 1008 1 4076KB/s 170K usec 1019 2 4655KB/s 291K usec 1163 1 2891KB/s 212K usec 722 4 5872KB/s 417K usec 1466 1 1881KB/s 411K usec 470 8 7312KB/s 841K usec 1824 1 853KB/s 816K usec 213 16 7844KB/s 1728K usec 1956 1 503KB/s 1609K usec 125 32 7920KB/s 3417K usec 1969 1 249KB/s 3205K usec 62
BW limit group1=5 MB/s BW limit group2=5 MB/s [Multiple Sequential Reader] [Sequential Reader] nr Agg-bandw Max-latency Agg-iops nr Agg-bandw Max-latency Agg-iops 1 2377KB/s 110K usec 594 1 2415KB/s 120K usec 603 2 2759KB/s 222K usec 689 1 1709KB/s 220K usec 427 4 3314KB/s 420K usec 828 1 1163KB/s 414K usec 290 8 4060KB/s 901K usec 1011 1 527KB/s 816K usec 131 16 4324KB/s 1613K usec 1074 1 311KB/s 1613K usec 77 32 4320KB/s 3235K usec 1067 1 163KB/s 3209K usec 40
Note that with bw limit 30MB/s, we are able to hit iops more than 3400 but with bw=5MB/s, we are hitting close to 1100 iops. So I think we are under-utilizing the storage here and not run into any kind of iops limit.
Quoted 11 lines
Quoted 11 lines
BW limit group1=10 MB/s BW limit group2=10 MB/s
[Multiple Sequential Reader] [Sequential Reader]nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 4128KB/s 4128KB/s 4032KB/s 215K usec 1 4076KB/s 170K usec 2 2880KB/s 1886KB/s 4655KB/s 291K usec 1 2891KB/s 212K usec 4 1912KB/s 888KB/s 5872KB/s 417K usec 1 1881KB/s 411K usec 8 1032KB/s 432KB/s 7312KB/s 841K usec 1 853KB/s 816K usec 16 540KB/s 259KB/s 7844KB/s 1728K usec 1 503KB/s 1609K usec 32 291KB/s 111KB/s 7920KB/s 3417K usec 1 249KB/s 3205K usec
Quoted 8 lines
Quoted 4 lines
Notes:
- Same test with 10MB/s as group limit. This is again a surprising result. Max BW in first group is being throttled but still throughput is dropping significantly in second group and latencies are on the rise.Same consideration about CFQ and/or iops limit. Could you post all the fio options you've used also for this test (or better, for all tests)?
Already posted in a separate mail.
Quoted 6 lines
Quoted 3 lines
- Limit of first group is 10MB/s but it is achieving max BW of around 8MB/s only. What happened to rest of the 2MB/s?
Ditto.
For 10MB/s case, max iops seems to be 2000 collectively, way below than 3400. So I doubt that this is case of hitting max iops.
Thanks
Vivek
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: Performance numbers with IO throttling patches (Was: Re: IO scheduler based IO controller V10)
Quoted 7 lines · Vivek Goyal
On Thu, Sep 24, 2009 at 02:33:15PM -0700, Andrew Morton wrote:
[..]
Quoted 4 lines
Quoted 3 lines
Environment
==========
A 7200 RPM SATA drive with queue depth of 31. Ext3 filesystem.
Quoted 8 lines
Quoted 6 lines
That's a bit of a toy.
Do we have testing results for more enterprisey hardware? Big storage arrays? SSD? Infiniband? iscsi? nfs? (lol, gotcha)
Hi All,
Hi Vivek,
thanks for posting this detailed report first of all. A few comments below.
Quoted 119 lines
Couple of days back I posted some performance number of "IO scheduler controller" and "dm-ioband" here.
http://lkml.org/lkml/2009/10/8/9
Now I have run similar tests with Andrea Righi's IO throttling approach of max bandwidth control. This is the exercise to understand pros/cons of each approach and see how can we take things forward.
Environment
===========
Software
--------
- 2.6.31 kenrel
- IO scheduler controller V10 on top of 2.6.31
- IO throttling patch on top of 2.6.31. Patch is available here.http://www.develer.com/~arighi/linux/patches/io-throttle/old/cgroup-io-throttle-2.6.31.patch
Hardware
--------
A storage array of 5 striped disks of 500GB each.Used fio jobs for 30 seconds in various configurations. Most of the IO is direct IO to eliminate the effects of caches.
I have run three sets for each test. Blindly reporting results of set2 from each test, otherwise it is too much of data to report.
Had lun of 2500GB capacity. Used 200G partition with ext3 file system for my testing. For IO scheduler controller testing, created two cgroups of weight 100 each so that effectively disk can be divided half/half between two groups.
For IO throttling patches also created two cgroups. Now tricky part is that it is a max bw controller and not a proportional weight controller. So dividing the disk capacity half/half between two cgroups is tricky. The reason being I just don't know what's the BW capacity of underlying storage. Throughput varies so much with type of workload. For example, on my arrary, this is how throughput looks like with different workloads.
8 sequential buffered readers 115 MB/s
8 direct sequential readers bs=64K 64 MB/s
8 direct sequential readers bs=4K 14 MB/s8 buffered random readers bs=64K 3 MB/s
8 direct random readers bs=64K 15 MB/s
8 direct random readers bs=4K 1.5 MB/sSo throughput seems to be varying from 1.5 MB/s to 115 MB/s depending on workload. What should be the BW limits per cgroup to divide disk BW in half/half between two groups?
So I took a conservative estimate and divide max bandwidth divide by 2, and thought of array capacity as 60MB/s and assign each cgroup 30MB/s. In some cases I have assigened even 10MB/s or 5MB/s to each cgropu to see the effects of throttling. I am using "Leaky bucket" policy for all the tests.
As theme of two controllers is different, at some places it might sound like apples vs oranges comparison. But still it does help...
Multiple Random Reader vs Sequential Reader
===============================================
Generally random readers bring the throughput down of others in the system. Ran a test to see the impact of increasing number of random readers on single sequential reader in different groups.Vanilla CFQ
-----------------------------------
[Multiple Random Reader] [Sequential Reader]nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 23KB/s 23KB/s 22KB/s 691 msec 1 13519KB/s 468K usec 2 152KB/s 152KB/s 297KB/s 244K usec 1 12380KB/s 31675 usec 4 174KB/s 156KB/s 638KB/s 249K usec 1 10860KB/s 36715 usec 8 49KB/s 11KB/s 310KB/s 1856 msec 1 1292KB/s 990K usec 16 63KB/s 48KB/s 877KB/s 762K usec 1 3905KB/s 506K usec 32 35KB/s 27KB/s 951KB/s 2655 msec 1 1109KB/s 1910K usecIO scheduler controller + CFQ
-----------------------------------
[Multiple Random Reader] [Sequential Reader]nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 228KB/s 228KB/s 223KB/s 132K usec 1 5551KB/s 129K usec 2 97KB/s 97KB/s 190KB/s 154K usec 1 5718KB/s 122K usec 4 115KB/s 110KB/s 445KB/s 208K usec 1 5909KB/s 116K usec 8 23KB/s 12KB/s 158KB/s 2820 msec 1 5445KB/s 168K usec 16 11KB/s 3KB/s 145KB/s 5963 msec 1 5418KB/s 164K usec 32 6KB/s 2KB/s 139KB/s 12762 msec 1 5398KB/s 175K usecNotes:
- Sequential reader in group2 seems to be well isolated from random readers in group1. Throughput and latency of sequential reader are stable and don't drop as number of random readers inrease in system.io-throttle + CFQ
------------------
BW limit group1=10 MB/s BW limit group2=10 MB/s
[Multiple Random Reader] [Sequential Reader]nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 37KB/s 37KB/s 36KB/s 218K usec 1 8006KB/s 20529 usec 2 185KB/s 183KB/s 360KB/s 228K usec 1 7475KB/s 33665 usec 4 188KB/s 171KB/s 699KB/s 262K usec 1 6800KB/s 46224 usec 8 84KB/s 51KB/s 573KB/s 1800K usec 1 2835KB/s 885K usec 16 21KB/s 9KB/s 294KB/s 3590 msec 1 437KB/s 1855K usec 32 34KB/s 27KB/s 980KB/s 2861K usec 1 1145KB/s 1952K usecNotes:
- I have setup limits of 10MB/s in both the cgroups. Now random reader group will never achieve that kind of speed, so it will not be throttled and then it goes onto impact the throughput and latency of other groups in the system.- Now the key question is how conservative one should in be setting up max BW limit. On this box if a customer has bought 10MB/s cgroup and if he is running some random readers it will kill throughput of other groups in the system and their latencies will shoot up. No isolation in this case.
Quoted 3 lines
- So in general, max BW provides isolation from high speed groups but it does not provide isolaton from random reader groups which are moving slow.
Remember that in addition to blockio.bandwidth-max the io-throttle controlller also provides blockio.iops-max to enforce hard limits on the number of IO operations per second. Probably for this testcase both cgroups should be limited in terms of BW and iops to achieve a better isolation.
Quoted 57 lines
Multiple Sequential Reader vs Random Reader
===============================================
Now running a reverse test where in one group I am running increasing number of sequential readers and in other group I am running one random reader and see the impact of sequential readers on random reader.Vanilla CFQ
-----------------------------------
[Multiple Sequential Reader] [Random Reader]nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 13978KB/s 13978KB/s 13650KB/s 27614 usec 1 22KB/s 227 msec 2 6225KB/s 6166KB/s 12101KB/s 568K usec 1 10KB/s 457 msec 4 4052KB/s 2462KB/s 13107KB/s 322K usec 1 6KB/s 841 msec 8 1899KB/s 557KB/s 12960KB/s 829K usec 1 13KB/s 1628 msec 16 1007KB/s 279KB/s 13833KB/s 1629K usec 1 10KB/s 3236 msec 32 506KB/s 98KB/s 13704KB/s 3389K usec 1 6KB/s 3238 msecIO scheduler controller + CFQ
-----------------------------------
[Multiple Sequential Reader] [Random Reader]nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 5721KB/s 5721KB/s 5587KB/s 126K usec 1 223KB/s 126K usec 2 3216KB/s 1442KB/s 4549KB/s 349K usec 1 224KB/s 176K usec 4 1895KB/s 640KB/s 5121KB/s 775K usec 1 222KB/s 189K usec 8 957KB/s 285KB/s 6368KB/s 1680K usec 1 223KB/s 142K usec 16 458KB/s 132KB/s 6455KB/s 3343K usec 1 219KB/s 165K usec 32 248KB/s 55KB/s 6001KB/s 6957K usec 1 220KB/s 504K usecNotes:
- Random reader is well isolated from increasing number of sequential readers in other group. BW and latencies are stable.io-throttle + CFQ
-----------------------------------
BW limit group1=10 MB/s BW limit group2=10 MB/s
[Multiple Sequential Reader] [Random Reader]nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 8200KB/s 8200KB/s 8007KB/s 20275 usec 1 37KB/s 217K usec 2 3926KB/s 3919KB/s 7661KB/s 122K usec 1 16KB/s 441 msec 4 2271KB/s 1497KB/s 7672KB/s 611K usec 1 9KB/s 927 msec 8 1113KB/s 513KB/s 7507KB/s 849K usec 1 21KB/s 1020 msec 16 661KB/s 236KB/s 7959KB/s 1679K usec 1 13KB/s 2926 msec 32 292KB/s 109KB/s 7864KB/s 3446K usec 1 8KB/s 3439 msecBW limit group1=5 MB/s BW limit group2=5 MB/s
[Multiple Sequential Reader] [Random Reader]nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 4686KB/s 4686KB/s 4576KB/s 21095 usec 1 57KB/s 219K usec 2 2298KB/s 2179KB/s 4372KB/s 132K usec 1 37KB/s 431K usec 4 1245KB/s 1019KB/s 4449KB/s 324K usec 1 26KB/s 835 msec 8 584KB/s 403KB/s 4109KB/s 833K usec 1 30KB/s 1625K usec 16 346KB/s 252KB/s 4605KB/s 1641K usec 1 129KB/s 3236K usec 32 175KB/s 56KB/s 4269KB/s 3236K usec 1 8KB/s 3235 msecNotes:
Quoted 7 lines
- Above result is surprising to me. I have run it twice. In first run, I setup per cgroup limit as 10MB/s and in second run I set it up 5MB/s. In both the cases as number of sequential readers increase in other groups, random reader's throughput decreases and latencies increase. This is happening despite the fact that sequential readers are being throttled to make sure it does not impact workload in other group. Wondering why random readers are not seeing consistent throughput and latencies.
Maybe because CFQ is still trying to be fair among processes instead of cgroups. Remember that io-throttle doesn't touch the CFQ code (for this I'm definitely convinced that CFQ should be changed to think also in terms of cgroups, and io-throttle alone is not enough).
So, even if group1 is being throttled in part it is still able to submit some requests that get a higher priority respect to the requests submitted by the single random reader task.
It could be interesting to test another IO scheduler (deadline, as or even noop) to check if this is the actual problem.
Quoted 6 lines
- Andrea, can you please also run similar tests to see if you see same results or not. This is to rule out any testing methodology errors or scripting bugs. :-). I also have collected the snapshot of some cgroup files like bandwidth-max, throttlecnt, and stats. Let me know if you want those to see what is happenig here.
Sure, I'll do some tests ASAP. Another interesting test would be to set a blockio.iops-max limit also for the sequential readers' cgroup, to be sure we're not touching some iops physical disk limit.
Could you post all the options you used with fio, so I can repeat some tests as similar as possible to yours?
Quoted 45 lines
Multiple Sequential Reader vs Sequential Reader
===============================================
- This time running random readers are out of the picture and trying to see the effect of increasing number of sequential readers on another sequential reader running in a different group.Vanilla CFQ
-----------------------------------
[Multiple Sequential Reader] [Sequential Reader]nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 6325KB/s 6325KB/s 6176KB/s 114K usec 1 6902KB/s 120K usec 2 4588KB/s 3102KB/s 7510KB/s 571K usec 1 4564KB/s 680K usec 4 3242KB/s 1158KB/s 9469KB/s 495K usec 1 3198KB/s 410K usec 8 1775KB/s 459KB/s 12011KB/s 1178K usec 1 1366KB/s 818K usec 16 943KB/s 296KB/s 13285KB/s 1923K usec 1 728KB/s 1816K usec 32 511KB/s 148KB/s 13555KB/s 3286K usec 1 391KB/s 3212K usecIO scheduler controller + CFQ
-----------------------------------
[Multiple Sequential Reader] [Sequential Reader]nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 6781KB/s 6781KB/s 6622KB/s 109K usec 1 6691KB/s 115K usec 2 3758KB/s 1876KB/s 5502KB/s 693K usec 1 6373KB/s 419K usec 4 2100KB/s 671KB/s 5751KB/s 987K usec 1 6330KB/s 569K usec 8 1023KB/s 355KB/s 6969KB/s 1569K usec 1 6086KB/s 120K usec 16 520KB/s 130KB/s 7094KB/s 3140K usec 1 5984KB/s 119K usec 32 245KB/s 86KB/s 6621KB/s 6571K usec 1 5850KB/s 113K usecNotes:
- BW and latencies of sequential reader in group 2 are fairly stable as number of readers increase in first group.io-throttle + CFQ
-----------------------------------
BW limit group1=30 MB/s BW limit group2=30 MB/s
[Multiple Sequential Reader] [Sequential Reader]nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 6343KB/s 6343KB/s 6195KB/s 116K usec 1 6993KB/s 109K usec 2 4583KB/s 3046KB/s 7451KB/s 583K usec 1 4516KB/s 433K usec 4 2945KB/s 1324KB/s 9552KB/s 602K usec 1 3001KB/s 583K usec 8 1804KB/s 473KB/s 12257KB/s 861K usec 1 1386KB/s 815K usec 16 942KB/s 265KB/s 13560KB/s 1659K usec 1 718KB/s 1658K usec 32 462KB/s 143KB/s 13757KB/s 3482K usec 1 409KB/s 3480K usec
Quoted 7 lines
Notes:
- BW decreases and latencies increase in group2 as number of readers increase in first group. This should be due to fact that no throttling will happen as none of the groups is hitting the limit of 30MB/s. To me this is the tricky part. How a service provider is supposed to set the limit of groups. If groups are not hitting max limits, it will still impact the BW and latencies in other group.
Are you using 4k block size here? because in case of too small blocks you could hit some physical iops limit. Also for this case it could be interesting to see what happens setting both BW and iops hard limits.
Quoted 11 lines
BW limit group1=10 MB/s BW limit group2=10 MB/s
[Multiple Sequential Reader] [Sequential Reader]nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 4128KB/s 4128KB/s 4032KB/s 215K usec 1 4076KB/s 170K usec 2 2880KB/s 1886KB/s 4655KB/s 291K usec 1 2891KB/s 212K usec 4 1912KB/s 888KB/s 5872KB/s 417K usec 1 1881KB/s 411K usec 8 1032KB/s 432KB/s 7312KB/s 841K usec 1 853KB/s 816K usec 16 540KB/s 259KB/s 7844KB/s 1728K usec 1 503KB/s 1609K usec 32 291KB/s 111KB/s 7920KB/s 3417K usec 1 249KB/s 3205K usec
Quoted 4 lines
Notes:
- Same test with 10MB/s as group limit. This is again a surprising result. Max BW in first group is being throttled but still throughput is dropping significantly in second group and latencies are on the rise.
Same consideration about CFQ and/or iops limit. Could you post all the fio options you've used also for this test (or better, for all tests)?
Quoted 3 lines
- Limit of first group is 10MB/s but it is achieving max BW of around 8MB/s only. What happened to rest of the 2MB/s?
Ditto.
Quoted 134 lines
- Andrea, again, please do run this test. The throughput drop in second group stumps me and forces me to think if I am doing something wrong.
BW limit group1=5 MB/s BW limit group2=5 MB/s
[Multiple Sequential Reader] [Sequential Reader]nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 2434KB/s 2434KB/s 2377KB/s 110K usec 1 2415KB/s 120K usec 2 1639KB/s 1186KB/s 2759KB/s 222K usec 1 1709KB/s 220K usec 4 1114KB/s 648KB/s 3314KB/s 420K usec 1 1163KB/s 414K usec 8 567KB/s 366KB/s 4060KB/s 901K usec 1 527KB/s 816K usec 16 329KB/s 179KB/s 4324KB/s 1613K usec 1 311KB/s 1613K usec 32 178KB/s 70KB/s 4320KB/s 3235K usec 1 163KB/s 3209K usec- Setting the limit to 5MB/s per group also does not seem to be helping the second group.
Multiple Random Writer vs Random Reader
===============================================
This time running multiple random writers in first group and see the impact on throughput and latency of random reader in different group.Vanilla CFQ
-----------------------------------
[Multiple Random Writer] [Random Reader]nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 64018KB/s 64018KB/s 62517KB/s 353K usec 1 190KB/s 96 msec 2 35298KB/s 35257KB/s 68899KB/s 208K usec 1 76KB/s 2416 msec 4 16387KB/s 14662KB/s 60630KB/s 3746K usec 1 106KB/s 2308K usec 8 5106KB/s 3492KB/s 33335KB/s 2995K usec 1 193KB/s 2292K usec 16 3676KB/s 3002KB/s 51807KB/s 2283K usec 1 72KB/s 2298K usec 32 2169KB/s 1480KB/s 56882KB/s 1990K usec 1 35KB/s 1093 msecIO scheduler controller + CFQ
-----------------------------------
[Multiple Random Writer] [Random Reader]nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 20369KB/s 20369KB/s 19892KB/s 877K usec 1 255KB/s 137K usec 2 14347KB/s 14288KB/s 27964KB/s 1010K usec 1 228KB/s 117K usec 4 6996KB/s 6701KB/s 26775KB/s 1362K usec 1 221KB/s 180K usec 8 2849KB/s 2770KB/s 22007KB/s 2660K usec 1 250KB/s 485K usec 16 1463KB/s 1365KB/s 22384KB/s 2606K usec 1 254KB/s 115K usec 32 799KB/s 681KB/s 22404KB/s 2879K usec 1 266KB/s 107K usecNotes
- BW and latencies of random reader in second group are fairly stable.io-throttle + CFQ
-----------------------------------
BW limit group1=30 MB/s BW limit group2=30 MB/s
[Multiple Random Writer] [Random Reader]nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 21920KB/s 21920KB/s 21406KB/s 1017K usec 1 353KB/s 432K usec 2 14291KB/s 9626KB/s 23357KB/s 1832K usec 1 362KB/s 177K usec 4 7130KB/s 5135KB/s 24736KB/s 1336K usec 1 348KB/s 425K usec 8 3165KB/s 2949KB/s 23792KB/s 2133K usec 1 336KB/s 146K usec 16 1653KB/s 1406KB/s 23694KB/s 2198K usec 1 337KB/s 115K usec 32 793KB/s 717KB/s 23198KB/s 2195K usec 1 330KB/s 192K usecBW limit group1=10 MB/s BW limit group2=10 MB/s
[Multiple Random Writer] [Random Reader]nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 7903KB/s 7903KB/s 7718KB/s 1037K usec 1 474KB/s 103K usec 2 4496KB/s 4428KB/s 8715KB/s 1091K usec 1 450KB/s 553K usec 4 2153KB/s 1827KB/s 7914KB/s 2042K usec 1 458KB/s 108K usec 8 1129KB/s 1087KB/s 8688KB/s 1280K usec 1 432KB/s 98215 usec 16 606KB/s 527KB/s 8668KB/s 2303K usec 1 426KB/s 90609 usec 32 312KB/s 259KB/s 8599KB/s 2557K usec 1 441KB/s 95283 usecNotes:
- IO throttling seems to be working really well here. Random writers are contained in the first group and this gives stable BW and latencies to random reader in second group.Multiple Buffered Writer vs Buffered Writer
===========================================
This time run multiple buffered writers in group1 and see run a single buffered writer in other group and see if we can provide fairness and isolation.Vanilla CFQ
------------
[Multiple Buffered Writer] [Buffered Writer]nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 68997KB/s 68997KB/s 67380KB/s 645K usec 1 67122KB/s 567K usec 2 47509KB/s 46218KB/s 91510KB/s 865K usec 1 45118KB/s 865K usec 4 28002KB/s 26906KB/s 105MB/s 1649K usec 1 26879KB/s 1643K usec 8 15985KB/s 14849KB/s 117MB/s 943K usec 1 15653KB/s 766K usec 16 11567KB/s 6881KB/s 128MB/s 1174K usec 1 7333KB/s 947K usec 32 5877KB/s 3649KB/s 130MB/s 1205K usec 1 5142KB/s 988K usecIO scheduler controller + CFQ
-----------------------------------
[Multiple Buffered Writer] [Buffered Writer]nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 68580KB/s 68580KB/s 66972KB/s 2901K usec 1 67194KB/s 2901K usec 2 47419KB/s 45700KB/s 90936KB/s 3149K usec 1 44628KB/s 2377K usec 4 27825KB/s 27274KB/s 105MB/s 1177K usec 1 27584KB/s 1177K usec 8 15382KB/s 14288KB/s 114MB/s 1539K usec 1 14794KB/s 783K usec 16 9161KB/s 7592KB/s 124MB/s 3177K usec 1 7713KB/s 886K usec 32 4928KB/s 3961KB/s 126MB/s 1152K usec 1 6465KB/s 4510K usecNotes:
- It does not work. Buffered writer in second group are being overwhelmed by writers in group1.- This is a limitation of IO scheduler based controller currently as page cache at higher layer evens out the traffic and does not throw more traffic from higher weight group.
- This is something needs more work at higher layers like dirty limts per cgroup in memory contoller and the method to writeout buffered pages belonging to a particular memory cgroup. This is still being brainstormed.
io-throttle + CFQ
-----------------------------------
BW limit group1=30 MB/s BW limit group2=30 MB/s
[Multiple Buffered Writer] [Buffered Writer]nr Max-bandw Min-bandw Agg-bandw Max-latency nr Agg-bandw Max-latency 1 33863KB/s 33863KB/s 33070KB/s 3046K usec 1 25165KB/s 13248K usec 2 13457KB/s 12906KB/s 25745KB/s 9286K usec 1 29958KB/s 3736K usec 4 7414KB/s 6543KB/s 27145KB/s 10557K usec 1 30968KB/s 8356K usec 8 3562KB/s 2640KB/s 24430KB/s 12012K usec 1 30801KB/s 7037K usec 16 3962KB/s 881KB/s 26632KB/s 12650K usec 1 31150KB/s 7173K usec 32 3275KB/s 406KB/s 27295KB/s 14609K usec 1 26328KB/s 8069K usecNotes:
- This seems to work well here. io-throttle is throttling the writers before they write too much of data in page cache. One side effect of this seems to be that now a process will not be allowed to write at memory speed in page cahce and will be limited to disk IO speed limits set for the cgroup.
Quoted 6 lines
Andrea is thinking of removing throttling in
balance_dirty_pages()to allow writting at disk speed till we hitdirty_limits. But removing it leads to a different issue where too many dirty pages from a single group can be present from a cgroup in page cache and if that cgroup is slow moving one, then pages are flushed to disk at slower speed delyaing other higher rate cgroups. (all discussed in private mails with Andrea).
I confirm this. :) But IMHO before removing the throttling in balance_dirty_pages() we really need the per-cgroup dirty limit / dirty page cache quota.
Quoted 17 lines
ioprio class and iopriority with-in cgroups issues with IO-throttle
===================================================================Currently throttling logic is designed in such a way that it makes the throttling uniform for every process in the group. So we will loose the differentiation between different class of processes or differnetitation between different priority of processes with-in group.
I have run the tests of these in the past and reported it here in the past.
https://lists.linux-foundation.org/pipermail/containers/2009-May/017588.html
Thanks
Vivek
Signature · 2 lines
Andrea Righi - Develer s.r.l http://www.develer.com
Quoted 3 lines · Vivek Goyal
Notes:
- With vanilla CFQ, random writers can overwhelm a random reader. Bring down its throughput and bump up latencies significantly.
IIRC, with vanilla CFQ, sequential writing can overwhelm random readers, too.
I'm basing this assumption on the observations I made on both OpenSuse 11.1 and Ubuntu 9.10 alpha6 which I described in my posting on LKML
titled: "Poor desktop responsiveness with background I/O-operations" of 2009-09-20.
(Message ID: [email protected])
Thus, I'm posting this to show that your work is greatly appreciated, given the rather disappointig status quo of Linux's fairness when it comes to disk IO time.
I hope that your efforts lead to a change in performance of current userland applications, the sooner, the better.
Thanks
Ulrich
Quoted 16 lines · Ulrich Lukas
Quoted 3 lines · Vivek Goyal
Notes:
- With vanilla CFQ, random writers can overwhelm a random reader. Bring down its throughput and bump up latencies significantly.IIRC, with vanilla CFQ, sequential writing can overwhelm random readers, too.
I'm basing this assumption on the observations I made on both OpenSuse 11.1 and Ubuntu 9.10 alpha6 which I described in my posting on LKML
titled: "Poor desktop responsiveness with background I/O-operations" of 2009-09-20.
(Message ID: [email protected])
Quoted 7 lines
Thus, I'm posting this to show that your work is greatly appreciated, given the rather disappointig status quo of Linux's fairness when it comes to disk IO time.
I hope that your efforts lead to a change in performance of current userland applications, the sooner, the better.
[Please don't remove people from original CC list. I am putting them back.]
Hi Ulrich,
I quicky went through that mail thread and I tried following on my desktop.
##########################################
dd if=/home/vgoyal/4G-file of=/dev/null &
sleep 5
time firefox
# close firefox once gui pops up. ##########################################
It was taking close to 1 minute 30 seconds to launch firefox and dd got following.
4294967296 bytes (4.3 GB) copied, 100.602 s, 42.7 MB/s
(Results do vary across runs, especially if system is booted fresh. Don't
know why...).
Then I tried putting both the applications in separate groups and assign them weights 200 each.
##########################################
dd if=/home/vgoyal/4G-file of=/dev/null &
echo $! > /cgroup/io/test1/tasks
sleep 5
echo $$ > /cgroup/io/test2/tasks
time firefox
# close firefox once gui pops up. ##########################################
Now I firefox pops up in 27 seconds. So it cut down the time by 2/3.
4294967296 bytes (4.3 GB) copied, 84.6138 s, 50.8 MB/s
Notice that throughput of dd also improved.
I ran the block trace and noticed in many a cases firefox threads immediately preempted the "dd". Probably because it was a file system request. So in this case latency will arise from seek time.
In some other cases, threads had to wait for up to 100ms because dd was not preempted. In this case latency will arise both from waiting on queue as well as seek time.
With cgroup thing, We will run 100ms slice for the group in which firefox is being launched and then give 100ms uninterrupted time slice to dd. So it should cut down on number of seeks happening and that's why we probably see this improvement.
So grouping can help in such cases. May be you can move your X session in one group and launch the big IO in other group. Most likely you should have better desktop experience without compromising on dd thread output.
Thanks
Vivek
Quoted 65 lines · Vivek Goyal
Quoted 16 lines · Ulrich Lukas
Quoted 3 lines · Vivek Goyal
Notes:
- With vanilla CFQ, random writers can overwhelm a random reader. Bring down its throughput and bump up latencies significantly.IIRC, with vanilla CFQ, sequential writing can overwhelm random readers, too.
I'm basing this assumption on the observations I made on both OpenSuse 11.1 and Ubuntu 9.10 alpha6 which I described in my posting on LKML
titled: "Poor desktop responsiveness with background I/O-operations" of 2009-09-20.
(Message ID: [email protected])Quoted 7 lines
Thus, I'm posting this to show that your work is greatly appreciated, given the rather disappointig status quo of Linux's fairness when it comes to disk IO time.
I hope that your efforts lead to a change in performance of current userland applications, the sooner, the better.
[Please don't remove people from original CC list. I am putting them back.]
Hi Ulrich,
I quicky went through that mail thread and I tried following on my desktop.
##########################################
dd if=/home/vgoyal/4G-file of=/dev/null &
sleep 5
time firefox# close firefox once gui pops up. ##########################################It was taking close to 1 minute 30 seconds to launch firefox and dd got following.
4294967296 bytes (4.3 GB) copied, 100.602 s, 42.7 MB/s
(Results do vary across runs, especially if system is booted fresh. Don't
know why...).Then I tried putting both the applications in separate groups and assign them weights 200 each.
##########################################
dd if=/home/vgoyal/4G-file of=/dev/null &
echo $! > /cgroup/io/test1/tasks
sleep 5
echo $$ > /cgroup/io/test2/tasks
time firefox# close firefox once gui pops up. ##########################################Now I firefox pops up in 27 seconds. So it cut down the time by 2/3.
4294967296 bytes (4.3 GB) copied, 84.6138 s, 50.8 MB/s
Notice that throughput of dd also improved.
Quoted 7 lines
I ran the block trace and noticed in many a cases firefox threads immediately preempted the "dd". Probably because it was a file system request. So in this case latency will arise from seek time.
In some other cases, threads had to wait for up to 100ms because dd was not preempted. In this case latency will arise both from waiting on queue as well as seek time.
Hm, with tip, I see ~10ms max wakeup latency running scriptlet below.
Quoted 4 lines
With cgroup thing, We will run 100ms slice for the group in which firefox is being launched and then give 100ms uninterrupted time slice to dd. So it should cut down on number of seeks happening and that's why we probably see this improvement.
I'm not testing with group IO/CPU, but my numbers kinda agree that it's seek latency that's THE killer. What the compiled numbers below from the cheezy script below that _seem_ to be telling me is that the default setting of CFQ quantum is allowing too many write requests through, inflicting too much read latency... for the disk where my binaries live. The longer the seeky burst, the more it hurts both reader/writer, so cutting down the max requests queueable helps the reader (which i think can't queue anything near per unit time that the writer can) finish and get out of the writer's way sooner.
'nuff possibly useless words, onward to possibly useless numbers :)
dd pre == number dd emits upon receiving USR1 before execing perf. perf stat == time to load/execute perf stat konsole -e exit. dd post == same after dd number, after perf finishes.
quantum = 1 Avg
dd pre 58.4 52.5 56.1 61.6 52.3 56.1 MB/s perf stat 2.87 0.91 1.64 1.41 0.90 1.5 Sec dd post 56.6 61.0 66.3 64.7 60.9 61.9
quantum = 2
dd pre 59.7 62.4 58.9 65.3 60.3 61.3 perf stat 5.81 6.09 6.24 10.13 6.21 6.8 dd post 64.0 62.6 64.2 60.4 61.1 62.4
quantum = 3
dd pre 65.5 57.7 54.5 51.1 56.3 57.0 perf stat 14.01 13.71 8.35 5.35 8.57 9.9 dd post 59.2 49.1 58.8 62.3 62.1 58.3
quantum = 4
dd pre 57.2 52.1 56.8 55.2 61.6 56.5 perf stat 11.98 1.61 9.63 16.21 11.13 10.1 dd post 57.2 52.6 62.2 49.3 50.2 54.3
Nothing pinned btw, 4 cores available, but only 1 drive.
#!/bin/sh
DISK=sdb
QUANTUM=/sys/block/$DISK/queue/iosched/quantum
END=$(cat $QUANTUM)
for q in seq 1 $END; do
echo $q > $QUANTUM
LOGFILE=quantum_log_$q
rm -f $LOGFILE
for i in seq 1 5; do
echo 2 > /proc/sys/vm/drop_caches
sh -c "dd if=/dev/zero of=./deleteme.dd 2>&1|tee -a $LOGFILE" & sleep 30
sh -c "echo quantum $(cat $QUANTUM) loop $i" 2>&1|tee -a $LOGFILE perf stat -- killlall -q get_stuf_into_ram >/dev/null 2>&1 sleep 1
killall -q -USR1 dd &
sleep 1
sh -c "perf stat -- konsole -e exit" 2>&1|tee -a $LOGFILE sleep 1
killall -q -USR1 dd &
sleep 5
killall -qw dd
rm -f ./deleteme.dd
sync
sh -c "echo" 2>&1|tee -a $LOGFILE
done;
done;
My dd vs load non-cached binary woes seem to be coming from backmerge.
#if 0 /*MIKEDIDIT sand in gearbox?*/
/*
* See if our hash lookup can find a potential backmerge.
*/
__rq = elv_rqhash_find(q, bio->bi_sector);
if (__rq && elv_rq_merge_ok(__rq, bio)) {
*req = __rq;
return ELEVATOR_BACK_MERGE;
}#endif
- = stock = 0
+ = /sys/block/sdb/queue/nomerges = 1
x = backmerge disabled
quantum = 1 Avg
dd pre 58.4 52.5 56.1 61.6 52.3 56.1- MB/s virgin/foo
59.6 54.4 53.0 56.1 58.6 56.3+ 1.003
53.8 56.6 54.7 50.7 59.3 55.0x .980
perf stat 2.87 0.91 1.64 1.41 0.90 1.5- Sec
2.61 1.14 1.45 1.43 1.47 1.6+ 1.066
1.07 1.19 1.20 1.24 1.37 1.2x .800
dd post 56.6 61.0 66.3 64.7 60.9 61.9-
54.0 59.3 61.1 58.3 58.9 58.3+ .941
54.3 60.2 59.6 60.6 60.3 59.0x .953quantum = 2
dd pre 59.7 62.4 58.9 65.3 60.3 61.3-
49.4 51.9 58.7 49.3 52.4 52.3+ .853
58.3 52.8 53.1 50.4 59.9 54.9x .895
perf stat 5.81 6.09 6.24 10.13 6.21 6.8-
2.48 2.10 3.23 2.29 2.31 2.4+ .352
2.09 2.73 1.72 1.96 1.83 2.0x .294
dd post 64.0 62.6 64.2 60.4 61.1 62.4-
52.9 56.2 49.6 51.3 51.2 52.2+ .836
54.7 60.9 56.0 54.0 55.4 56.2x .900quantum = 3
dd pre 65.5 57.7 54.5 51.1 56.3 57.0-
58.1 53.9 52.2 58.2 51.8 54.8+ .961
60.5 56.5 56.7 55.3 54.6 56.7x .994
perf stat 14.01 13.71 8.35 5.35 8.57 9.9-
1.84 2.30 2.14 2.10 2.45 2.1+ .212
2.12 1.63 2.54 2.23 2.29 2.1x .212
dd post 59.2 49.1 58.8 62.3 62.1 58.3-
59.8 53.2 55.2 50.9 53.7 54.5+ .934
56.1 61.9 51.9 54.3 53.1 55.4x .950quantun = 4
dd pre 57.2 52.1 56.8 55.2 61.6 56.5-
48.7 55.4 51.3 49.7 54.5 51.9+ .918
55.8 54.5 50.3 56.4 49.3 53.2x .941
perf stat 11.98 1.61 9.63 16.21 11.13 10.1-
2.29 1.94 2.68 2.46 2.45 2.3+ .227
3.01 1.84 2.11 2.27 2.30 2.3x .227
dd post 57.2 52.6 62.2 49.3 50.2 54.3-
50.1 54.5 58.4 54.1 49.0 53.2+ .979
52.9 53.2 50.6 53.2 50.5 52.0x .957Quoted 11 lines · Mike Galbraith
My dd vs load non-cached binary woes seem to be coming from backmerge.
#if 0 /*MIKEDIDIT sand in gearbox?*/ /* * See if our hash lookup can find a potential backmerge. */ __rq = elv_rqhash_find(q, bio->bi_sector); if (__rq && elv_rq_merge_ok(__rq, bio)) { *req = __rq; return ELEVATOR_BACK_MERGE; }
Quoted 1 line
#endif
It's a given that not merging will provide better latency. We can't disable that or performance will suffer A LOT on some systems. There are ways to make it better, though. One would be to make the max request size smaller, but that would also hurt for streamed workloads. Can you try whether the below patch makes a difference? It will basically disallow merges to a request that isn't the last one.
We should probably make the merging logic a bit more clever, since the below wont work well for two (or more) streamed cases. I'll think a bit about that.
Note this is totally untested!
block/elevator.c +11 -3
@@ -497,9 +497,17 @@ int elv_merge(struct request_queue *q, struct request **req, struct bio *bio)* See if our hash lookup can find a potential backmerge. */ __rq =elv_rqhash_find (q, bio->bi_sector); -if (__rq &&elv_rq_merge_ok (__rq, bio)) { - *req = __rq; -return ELEVATOR_BACK_MERGE ; +if (__rq) { +/* +* If requests are queued behind this one, disallow merge. This +* prevents streaming IO from continually passing new IO. +*/ +if (elv_latter_request (q, __rq)) +return ELEVATOR_NO_MERGE ; +if (elv_rq_merge_ok (__rq, bio)) { + *req = __rq; +return ELEVATOR_BACK_MERGE ; + } }if (e->ops->elevator_merge_fn)
Signature · 1 line
Jens Axboe
Quoted 14 lines · Jens Axboe
Quoted 11 lines · Mike Galbraith
My dd vs load non-cached binary woes seem to be coming from backmerge.
#if 0 /*MIKEDIDIT sand in gearbox?*/ /* * See if our hash lookup can find a potential backmerge. */ __rq = elv_rqhash_find(q, bio->bi_sector); if (__rq && elv_rq_merge_ok(__rq, bio)) { *req = __rq; return ELEVATOR_BACK_MERGE; }Quoted 1 line
#endif
Quoted 1 line
It's a given that not merging will provide better latency.
Yeah, absolutely everything I've diddled that reduces the size of queued data improves the situation, which makes perfect sense. This one was a bit unexpected. Front merges didn't hurt at all, back merges did, and lots. After diddling the code a bit, I had the "well _duh_" moment.
Quoted 6 lines
We can't
disable that or performance will suffer A LOT on some systems. There are ways to make it better, though. One would be to make the max request size smaller, but that would also hurt for streamed workloads. Can you try whether the below patch makes a difference? It will basically disallow merges to a request that isn't the last one.
That's what all the looking I've done ends up at. Either you let the disk be all it can be, and you pay in latency, or you don't, and you pay in throughput.
Quoted 2 lines
below wont work well for two (or more) streamed cases. I'll think a bit about that.
Cool, think away. I've been eyeballing and pondering how to know when latency is going to become paramount. Absolutely nothing is happening, even for "it's my root".
Quoted 1 line
Note this is totally untested!
I'll give it a shot first thing in the A.M.
Note: I tested my stable of kernels today (22->), and we are better off dd vs read today than ever in this time span at least.
(i can't recall ever seeing a system where beating snot outta root didn't hurt really bad... would be very nice though;)
Quoted 26 lines
block/elevator.c +11
-3@@ -497,9 +497,17 @@ int elv_merge(struct request_queue *q, struct request **req, struct bio *bio)* See if our hash lookup can find a potential backmerge. */ __rq =elv_rqhash_find (q, bio->bi_sector); -if (__rq &&elv_rq_merge_ok (__rq, bio)) { - *req = __rq; -return ELEVATOR_BACK_MERGE ; +if (__rq) { +/* +* If requests are queued behind this one, disallow merge. This +* prevents streaming IO from continually passing new IO. +*/ +if (elv_latter_request (q, __rq)) +return ELEVATOR_NO_MERGE ; +if (elv_rq_merge_ok (__rq, bio)) { + *req = __rq; +return ELEVATOR_BACK_MERGE ; + } }if (e->ops->elevator_merge_fn)
Quoted 28 lines · Mike Galbraith
On Sun, 2009-09-27 at 18:42 +0200, Jens Axboe wrote:
I'll give it a shot first thing in the A.M.
Quoted 24 lines
block/elevator.c +11
-3@@ -497,9 +497,17 @@ int elv_merge(struct request_queue *q, struct request **req, struct bio *bio)* See if our hash lookup can find a potential backmerge. */ __rq =elv_rqhash_find (q, bio->bi_sector); -if (__rq &&elv_rq_merge_ok (__rq, bio)) { - *req = __rq; -return ELEVATOR_BACK_MERGE ; +if (__rq) { +/* +* If requests are queued behind this one, disallow merge. This +* prevents streaming IO from continually passing new IO. +*/ +if (elv_latter_request (q, __rq)) +return ELEVATOR_NO_MERGE ; +if (elv_rq_merge_ok (__rq, bio)) { + *req = __rq; +return ELEVATOR_BACK_MERGE ; + } }
Quoted 1 line
block/elevator.c int elv_merge(struct request_queue *q, struct request **req, struct bio *bio) · line 513
Quoted 1 line
block/elevator.c +0
-0if (e->ops->elevator_merge_fn)
- = virgin tip v2.6.31-10215-ga3c9602
+ = with patchlet
Avg
dd pre 67.4 70.9 65.4 68.9 66.2 67.7-
65.9 68.5 69.8 65.2 65.8 67.0- Avg
70.4 70.3 65.1 66.4 70.1 68.4- 67.7-
73.1 64.6 65.3 65.3 64.9 66.6+ 65.6+ .968
63.8 67.9 65.2 65.1 64.4 65.2+
64.9 66.3 64.1 65.2 64.8 65.0+
perf stat 8.66 16.29 9.65 14.88 9.45 11.7-
15.36 9.71 15.47 10.44 12.93 12.7-
10.55 15.11 10.22 15.35 10.32 12.3- 12.2-
9.87 7.53 10.62 7.51 9.95 9.0+ 9.1+ .745
7.73 10.12 8.19 11.87 8.07 9.1+
11.04 7.62 10.14 8.13 10.23 9.4+
dd post 63.4 60.5 66.7 64.5 67.3 64.4-
64.4 66.8 64.3 61.5 62.0 63.8-
63.8 64.9 66.2 65.6 66.9 65.4- 64.5-
60.9 63.4 60.2 63.4 65.5 62.6+ 61.8+ .958
63.3 59.9 61.9 62.7 61.2 61.8+
60.1 63.7 59.5 61.5 60.6 61.0+P.S.
Quoted 13 lines · Mike Galbraith
- = virgin tip v2.6.31-10215-ga3c9602
+ = with patchlet
Avgdd pre 67.4 70.9 65.4 68.9 66.2 67.7- 65.9 68.5 69.8 65.2 65.8 67.0- Avg 70.4 70.3 65.1 66.4 70.1 68.4- 67.7- 73.1 64.6 65.3 65.3 64.9 66.6+ 65.6+ .968 63.8 67.9 65.2 65.1 64.4 65.2+ 64.9 66.3 64.1 65.2 64.8 65.0+ perf stat 8.66 16.29 9.65 14.88 9.45 11.7- 15.36 9.71 15.47 10.44 12.93 12.7- 10.55 15.11 10.22 15.35 10.32 12.3- 12.2- 9.87 7.53 10.62 7.51 9.95 9.0+ 9.1+ .745
Quoted 8 lines
7.73 10.12 8.19 11.87 8.07 9.1+
11.04 7.62 10.14 8.13 10.23 9.4+
dd post 63.4 60.5 66.7 64.5 67.3 64.4-
64.4 66.8 64.3 61.5 62.0 63.8-
63.8 64.9 66.2 65.6 66.9 65.4- 64.5-
60.9 63.4 60.2 63.4 65.5 62.6+ 61.8+ .958
63.3 59.9 61.9 62.7 61.2 61.8+
60.1 63.7 59.5 61.5 60.6 61.0+Deadline and noop fsc^W are less than wonderful choices for this load.
perf stat 12.82 7.19 8.49 5.76 9.32 anticipatory
16.24 175.82 154.38 228.97 147.16 noop
43.23 57.39 96.13 148.25 180.09 deadline
28.65 167.40 195.95 183.69 178.61 deadline v2.6.27.35-Mike
A message that is not in the archive (sent off-list or not yet received): [email protected]
P.S.
Quoted 13 lines · Mike Galbraith
- = virgin tip v2.6.31-10215-ga3c9602
+ = with patchlet
Avgdd pre 67.4 70.9 65.4 68.9 66.2 67.7- 65.9 68.5 69.8 65.2 65.8 67.0- Avg 70.4 70.3 65.1 66.4 70.1 68.4- 67.7- 73.1 64.6 65.3 65.3 64.9 66.6+ 65.6+ .968 63.8 67.9 65.2 65.1 64.4 65.2+ 64.9 66.3 64.1 65.2 64.8 65.0+ perf stat 8.66 16.29 9.65 14.88 9.45 11.7- 15.36 9.71 15.47 10.44 12.93 12.7- 10.55 15.11 10.22 15.35 10.32 12.3- 12.2- 9.87 7.53 10.62 7.51 9.95 9.0+ 9.1+ .745
Quoted 8 lines
7.73 10.12 8.19 11.87 8.07 9.1+
11.04 7.62 10.14 8.13 10.23 9.4+
dd post 63.4 60.5 66.7 64.5 67.3 64.4-
64.4 66.8 64.3 61.5 62.0 63.8-
63.8 64.9 66.2 65.6 66.9 65.4- 64.5-
60.9 63.4 60.2 63.4 65.5 62.6+ 61.8+ .958
63.3 59.9 61.9 62.7 61.2 61.8+
60.1 63.7 59.5 61.5 60.6 61.0+Deadline and noop fsc^W are less than wonderful choices for this load.
perf stat 12.82 7.19 8.49 5.76 9.32 anticipatory
16.24 175.82 154.38 228.97 147.16 noop
43.23 57.39 96.13 148.25 180.09 deadline
28.65 167.40 195.95 183.69 178.61 deadline v2.6.27.35-Mike
Quoted 44 lines · Mike Galbraith
Quoted 28 lines · Mike Galbraith
On Sun, 2009-09-27 at 18:42 +0200, Jens Axboe wrote:
I'll give it a shot first thing in the A.M.
Quoted 24 lines
block/elevator.c +11
-3@@ -497,9 +497,17 @@ int elv_merge(struct request_queue *q, struct request **req, struct bio *bio)* See if our hash lookup can find a potential backmerge. */ __rq =elv_rqhash_find (q, bio->bi_sector); -if (__rq &&elv_rq_merge_ok (__rq, bio)) { - *req = __rq; -return ELEVATOR_BACK_MERGE ; +if (__rq) { +/* +* If requests are queued behind this one, disallow merge. This +* prevents streaming IO from continually passing new IO. +*/ +if (elv_latter_request (q, __rq)) +return ELEVATOR_NO_MERGE ; +if (elv_rq_merge_ok (__rq, bio)) { + *req = __rq; +return ELEVATOR_BACK_MERGE ; + } }Quoted 1 line
block/elevator.c int elv_merge(struct request_queue *q, struct request **req, struct bio *bio) · line 513
Quoted 1 line
block/elevator.c +0
-0if (e->ops->elevator_merge_fn)- = virgin tip v2.6.31-10215-ga3c9602
+ = with patchlet
Avgdd pre 67.4 70.9 65.4 68.9 66.2 67.7- 65.9 68.5 69.8 65.2 65.8 67.0- Avg 70.4 70.3 65.1 66.4 70.1 68.4- 67.7- 73.1 64.6 65.3 65.3 64.9 66.6+ 65.6+ .968 63.8 67.9 65.2 65.1 64.4 65.2+ 64.9 66.3 64.1 65.2 64.8 65.0+ perf stat 8.66 16.29 9.65 14.88 9.45 11.7- 15.36 9.71 15.47 10.44 12.93 12.7- 10.55 15.11 10.22 15.35 10.32 12.3- 12.2- 9.87 7.53 10.62 7.51 9.95 9.0+ 9.1+ .745
Quoted 9 lines
7.73 10.12 8.19 11.87 8.07 9.1+
11.04 7.62 10.14 8.13 10.23 9.4+
dd post 63.4 60.5 66.7 64.5 67.3 64.4-
64.4 66.8 64.3 61.5 62.0 63.8-
63.8 64.9 66.2 65.6 66.9 65.4- 64.5-
60.9 63.4 60.2 63.4 65.5 62.6+ 61.8+ .958
63.3 59.9 61.9 62.7 61.2 61.8+
60.1 63.7 59.5 61.5 60.6 61.0+Hmm.., so close to 25% reduction on average in completion time of konsole. But this is in presece of writer. Does this help even in presence of 1 or more sequential readers going?
So here latency seems to be coming from three sources.
- Wait in CFQ before request is dispatched (only in case of competing seq readers).
- seek latencies
- latencies because of bigger requests are already dispatched to disk.
So limiting the size of request will help with third factor but not with first two factors and here seek latencies seem to be the biggest contributor.
Thanks
Vivek
Quoted 44 lines · Mike Galbraith
Quoted 28 lines · Mike Galbraith
On Sun, 2009-09-27 at 18:42 +0200, Jens Axboe wrote:
I'll give it a shot first thing in the A.M.
Quoted 24 lines
block/elevator.c +11
-3@@ -497,9 +497,17 @@ int elv_merge(struct request_queue *q, struct request **req, struct bio *bio)* See if our hash lookup can find a potential backmerge. */ __rq =elv_rqhash_find (q, bio->bi_sector); -if (__rq &&elv_rq_merge_ok (__rq, bio)) { - *req = __rq; -return ELEVATOR_BACK_MERGE ; +if (__rq) { +/* +* If requests are queued behind this one, disallow merge. This +* prevents streaming IO from continually passing new IO. +*/ +if (elv_latter_request (q, __rq)) +return ELEVATOR_NO_MERGE ; +if (elv_rq_merge_ok (__rq, bio)) { + *req = __rq; +return ELEVATOR_BACK_MERGE ; + } }Quoted 1 line
block/elevator.c int elv_merge(struct request_queue *q, struct request **req, struct bio *bio) · line 513
Quoted 1 line
block/elevator.c +0
-0if (e->ops->elevator_merge_fn)- = virgin tip v2.6.31-10215-ga3c9602
+ = with patchlet
Avgdd pre 67.4 70.9 65.4 68.9 66.2 67.7- 65.9 68.5 69.8 65.2 65.8 67.0- Avg 70.4 70.3 65.1 66.4 70.1 68.4- 67.7- 73.1 64.6 65.3 65.3 64.9 66.6+ 65.6+ .968 63.8 67.9 65.2 65.1 64.4 65.2+ 64.9 66.3 64.1 65.2 64.8 65.0+ perf stat 8.66 16.29 9.65 14.88 9.45 11.7- 15.36 9.71 15.47 10.44 12.93 12.7- 10.55 15.11 10.22 15.35 10.32 12.3- 12.2- 9.87 7.53 10.62 7.51 9.95 9.0+ 9.1+ .745
Quoted 9 lines
7.73 10.12 8.19 11.87 8.07 9.1+
11.04 7.62 10.14 8.13 10.23 9.4+
dd post 63.4 60.5 66.7 64.5 67.3 64.4-
64.4 66.8 64.3 61.5 62.0 63.8-
63.8 64.9 66.2 65.6 66.9 65.4- 64.5-
60.9 63.4 60.2 63.4 65.5 62.6+ 61.8+ .958
63.3 59.9 61.9 62.7 61.2 61.8+
60.1 63.7 59.5 61.5 60.6 61.0+Hmm.., so close to 25% reduction on average in completion time of konsole. But this is in presece of writer. Does this help even in presence of 1 or more sequential readers going?
So here latency seems to be coming from three sources.
- Wait in CFQ before request is dispatched (only in case of competing seq readers).
- seek latencies
- latencies because of bigger requests are already dispatched to disk.
So limiting the size of request will help with third factor but not with first two factors and here seek latencies seem to be the biggest contributor.
Thanks
Vivek
Quoted 3 lines · Vivek Goyal
Hmm.., so close to 25% reduction on average in completion time of konsole. But this is in presece of writer. Does this help even in presence of 1 or more sequential readers going?
Dunno, I've only tested sequential writer.
Quoted 8 lines
So here latency seems to be coming from three sources.
- Wait in CFQ before request is dispatched (only in case of competing seq readers).
- seek latencies
- latencies because of bigger requests are already dispatched to disk.So limiting the size of request will help with third factor but not with first two factors and here seek latencies seem to be the biggest contributor.
Yeah, seek latency seems to dominate.
-Mike
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 3 lines · Vivek Goyal
Hmm.., so close to 25% reduction on average in completion time of konsole. But this is in presece of writer. Does this help even in presence of 1 or more sequential readers going?
Dunno, I've only tested sequential writer.
Quoted 8 lines
So here latency seems to be coming from three sources.
- Wait in CFQ before request is dispatched (only in case of competing seq readers).
- seek latencies
- latencies because of bigger requests are already dispatched to disk.So limiting the size of request will help with third factor but not with first two factors and here seek latencies seem to be the biggest contributor.
Yeah, seek latency seems to dominate.
-Mike
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 28 lines · Mike Galbraith
On Sun, 2009-09-27 at 18:42 +0200, Jens Axboe wrote:
I'll give it a shot first thing in the A.M.
Quoted 24 lines
block/elevator.c +11
-3@@ -497,9 +497,17 @@ int elv_merge(struct request_queue *q, struct request **req, struct bio *bio)* See if our hash lookup can find a potential backmerge. */ __rq =elv_rqhash_find (q, bio->bi_sector); -if (__rq &&elv_rq_merge_ok (__rq, bio)) { - *req = __rq; -return ELEVATOR_BACK_MERGE ; +if (__rq) { +/* +* If requests are queued behind this one, disallow merge. This +* prevents streaming IO from continually passing new IO. +*/ +if (elv_latter_request (q, __rq)) +return ELEVATOR_NO_MERGE ; +if (elv_rq_merge_ok (__rq, bio)) { + *req = __rq; +return ELEVATOR_BACK_MERGE ; + } }
Quoted 1 line
block/elevator.c int elv_merge(struct request_queue *q, struct request **req, struct bio *bio) · line 513
Quoted 1 line
block/elevator.c +0
-0if (e->ops->elevator_merge_fn)
- = virgin tip v2.6.31-10215-ga3c9602
+ = with patchlet
Avg
dd pre 67.4 70.9 65.4 68.9 66.2 67.7-
65.9 68.5 69.8 65.2 65.8 67.0- Avg
70.4 70.3 65.1 66.4 70.1 68.4- 67.7-
73.1 64.6 65.3 65.3 64.9 66.6+ 65.6+ .968
63.8 67.9 65.2 65.1 64.4 65.2+
64.9 66.3 64.1 65.2 64.8 65.0+
perf stat 8.66 16.29 9.65 14.88 9.45 11.7-
15.36 9.71 15.47 10.44 12.93 12.7-
10.55 15.11 10.22 15.35 10.32 12.3- 12.2-
9.87 7.53 10.62 7.51 9.95 9.0+ 9.1+ .745
7.73 10.12 8.19 11.87 8.07 9.1+
11.04 7.62 10.14 8.13 10.23 9.4+
dd post 63.4 60.5 66.7 64.5 67.3 64.4-
64.4 66.8 64.3 61.5 62.0 63.8-
63.8 64.9 66.2 65.6 66.9 65.4- 64.5-
60.9 63.4 60.2 63.4 65.5 62.6+ 61.8+ .958
63.3 59.9 61.9 62.7 61.2 61.8+
60.1 63.7 59.5 61.5 60.6 61.0+A message that is not in the archive (sent off-list or not yet received): 20090927164235.GA23126-tSWWG44O7X1aa/[email protected]
Quoted 14 lines · Jens Axboe
Quoted 11 lines · Mike Galbraith
My dd vs load non-cached binary woes seem to be coming from backmerge.
#if 0 /*MIKEDIDIT sand in gearbox?*/ /* * See if our hash lookup can find a potential backmerge. */ __rq = elv_rqhash_find(q, bio->bi_sector); if (__rq && elv_rq_merge_ok(__rq, bio)) { *req = __rq; return ELEVATOR_BACK_MERGE; }Quoted 1 line
#endif
Quoted 1 line
It's a given that not merging will provide better latency.
Yeah, absolutely everything I've diddled that reduces the size of queued data improves the situation, which makes perfect sense. This one was a bit unexpected. Front merges didn't hurt at all, back merges did, and lots. After diddling the code a bit, I had the "well _duh_" moment.
Quoted 6 lines
We can't
disable that or performance will suffer A LOT on some systems. There are ways to make it better, though. One would be to make the max request size smaller, but that would also hurt for streamed workloads. Can you try whether the below patch makes a difference? It will basically disallow merges to a request that isn't the last one.
That's what all the looking I've done ends up at. Either you let the disk be all it can be, and you pay in latency, or you don't, and you pay in throughput.
Quoted 2 lines
below wont work well for two (or more) streamed cases. I'll think a bit about that.
Cool, think away. I've been eyeballing and pondering how to know when latency is going to become paramount. Absolutely nothing is happening, even for "it's my root".
Quoted 1 line
Note this is totally untested!
I'll give it a shot first thing in the A.M.
Note: I tested my stable of kernels today (22->), and we are better off dd vs read today than ever in this time span at least.
(i can't recall ever seeing a system where beating snot outta root didn't hurt really bad... would be very nice though;)
Quoted 26 lines
block/elevator.c +11
-3@@ -497,9 +497,17 @@ int elv_merge(struct request_queue *q, struct request **req, struct bio *bio)* See if our hash lookup can find a potential backmerge. */ __rq =elv_rqhash_find (q, bio->bi_sector); -if (__rq &&elv_rq_merge_ok (__rq, bio)) { - *req = __rq; -return ELEVATOR_BACK_MERGE ; +if (__rq) { +/* +* If requests are queued behind this one, disallow merge. This +* prevents streaming IO from continually passing new IO. +*/ +if (elv_latter_request (q, __rq)) +return ELEVATOR_NO_MERGE ; +if (elv_rq_merge_ok (__rq, bio)) { + *req = __rq; +return ELEVATOR_BACK_MERGE ; + } }if (e->ops->elevator_merge_fn)
Quoted 6 lines · Jens Axboe
It's a given that not merging will provide better latency. We can't disable that or performance will suffer A LOT on some systems. There are ways to make it better, though. One would be to make the max request size smaller, but that would also hurt for streamed workloads. Can you try whether the below patch makes a difference? It will basically disallow merges to a request that isn't the last one.
Thoughts about something like the below?
The problem with the dd vs konsole -e exit type load seems to be kjournald overloading the disk between reads. When userland is blocked, kjournald is free to stuff 4*quantum into the queue instantly.
Taking the hint from Vivek's fairness tweakable patch, I stamped the queue when a seeker was last seen, and disallowed overload within CIC_SEEK_THR of that time. Worked well.
dd competing against perf stat -- konsole -e exec timings, 5 back to back runs
Avg
before 9.15 14.51 9.39 15.06 9.90 11.6 after 1.76 1.54 1.93 1.88 1.56 1.7
block/cfq-iosched.c +16 -3
@@ -174,6 +174,8 @@ struct cfq_data {unsigned int cfq_slice_async_rq;unsigned int cfq_slice_idle; +unsigned long last_seeker; +struct list_head cic_list;/*
@@ -1326,6 +1328,12 @@ static int cfq_dispatch_requests(struct request_queue *q, int force)return 0 ;/* +* We may have seeky queues, don't throttle up just yet. +*/ +if (time_before (jiffies, cfqd->last_seeker +CIC_SEEK_THR )) +return 0 ; + +/* * we are the only queue, allow up to 4 times of 'quantum' */ if (cfqq->dispatched >=4 * max_dispatch)
@@ -1941,7 +1949,7 @@ static void cfq_update_idle_window(struct cfq_data *cfqd, struct cfq_queue *cfqq, struct cfq_io_context *cic) { - int old_idle, enable_idle; + int old_idle, enable_idle, seeky = 0;/* * Don't idle for async or idle io prio class
@@ -1951,8 +1959,12 @@ cfq_update_idle_window(struct cfq_data *cfqd, struct cfq_queue *cfqq, enable_idle = old_idle =cfq_cfqq_idle_window (cfqq); -if (!atomic_read (&cic->ioc->nr_tasks) || !cfqd->cfq_slice_idle || - (cfqd->hw_tag &&CIC_SEEKY (cic))) +if (cfqd->hw_tag &&CIC_SEEKY (cic)) { + cfqd->last_seeker = jiffies; + seeky =1 ; + } + +if (!atomic_read (&cic->ioc->nr_tasks) || !cfqd->cfq_slice_idle || seeky) enable_idle =0 ;else if (sample_valid (cic->ttime_samples)) {if (cic->ttime_mean > cfqd->cfq_slice_idle)
@@ -2482,6 +2494,7 @@ static void *cfq_init_queue(struct request_queue *q) cfqd->cfq_slice_async_rq = cfq_slice_async_rq; cfqd->cfq_slice_idle = cfq_slice_idle; cfqd->hw_tag =1 ; + cfqd->last_seeker = jiffies;return cfqd; }
Quoted 6 lines · Jens Axboe
It's a given that not merging will provide better latency. We can't disable that or performance will suffer A LOT on some systems. There are ways to make it better, though. One would be to make the max request size smaller, but that would also hurt for streamed workloads. Can you try whether the below patch makes a difference? It will basically disallow merges to a request that isn't the last one.
Thoughts about something like the below?
The problem with the dd vs konsole -e exit type load seems to be kjournald overloading the disk between reads. When userland is blocked, kjournald is free to stuff 4*quantum into the queue instantly.
Taking the hint from Vivek's fairness tweakable patch, I stamped the queue when a seeker was last seen, and disallowed overload within CIC_SEEK_THR of that time. Worked well.
dd competing against perf stat -- konsole -e exec timings, 5 back to back runs
Avg
before 9.15 14.51 9.39 15.06 9.90 11.6 after 1.76 1.54 1.93 1.88 1.56 1.7
block/cfq-iosched.c +16 -3
@@ -174,6 +174,8 @@ struct cfq_data {unsigned int cfq_slice_async_rq;unsigned int cfq_slice_idle; +unsigned long last_seeker; +struct list_head cic_list;/*
@@ -1326,6 +1328,12 @@ static int cfq_dispatch_requests(struct request_queue *q, int force)return 0 ;/* +* We may have seeky queues, don't throttle up just yet. +*/ +if (time_before (jiffies, cfqd->last_seeker +CIC_SEEK_THR )) +return 0 ; + +/* * we are the only queue, allow up to 4 times of 'quantum' */ if (cfqq->dispatched >=4 * max_dispatch)
@@ -1941,7 +1949,7 @@ static void cfq_update_idle_window(struct cfq_data *cfqd, struct cfq_queue *cfqq, struct cfq_io_context *cic) { - int old_idle, enable_idle; + int old_idle, enable_idle, seeky = 0;/* * Don't idle for async or idle io prio class
@@ -1951,8 +1959,12 @@ cfq_update_idle_window(struct cfq_data *cfqd, struct cfq_queue *cfqq, enable_idle = old_idle =cfq_cfqq_idle_window (cfqq); -if (!atomic_read (&cic->ioc->nr_tasks) || !cfqd->cfq_slice_idle || - (cfqd->hw_tag &&CIC_SEEKY (cic))) +if (cfqd->hw_tag &&CIC_SEEKY (cic)) { + cfqd->last_seeker = jiffies; + seeky =1 ; + } + +if (!atomic_read (&cic->ioc->nr_tasks) || !cfqd->cfq_slice_idle || seeky) enable_idle =0 ;else if (sample_valid (cic->ttime_samples)) {if (cic->ttime_mean > cfqd->cfq_slice_idle)
@@ -2482,6 +2494,7 @@ static void *cfq_init_queue(struct request_queue *q) cfqd->cfq_slice_async_rq = cfq_slice_async_rq; cfqd->cfq_slice_idle = cfq_slice_idle; cfqd->hw_tag =1 ; + cfqd->last_seeker = jiffies;return cfqd; }
block/cfq-iosched.c static int cfq_dispatch_requests(struct request_queue *q, int force) · line 1335
Quoted 8 lines
block/cfq-iosched.c +5
-0/* +* We may have seeky queues, don't throttle up just yet. +*/ +if (time_before (jiffies, cfqd->last_seeker +CIC_SEEK_THR )) +return 0 ; +
bzzzt. Window too large, but the though is to let them overload, but not instantly.
-Mike
Quoted 12 lines · Mike Galbraith
block/cfq-iosched.c static int cfq_dispatch_requests(struct request_queue *q, int force) · line 1335
Quoted 8 lines
block/cfq-iosched.c +5
-0/* +* We may have seeky queues, don't throttle up just yet. +*/ +if (time_before (jiffies, cfqd->last_seeker +CIC_SEEK_THR )) +return 0 ; +bzzzt. Window too large, but the though is to let them overload, but not instantly.
CIC_SEEK_THR is 8K jiffies so that would be 8seconds on 1000HZ system. Try using one "slice_idle" period of 8 ms. But it might turn out to be too short depending on the disk speed.
Thanks
Vivek
Quoted 14 lines · Vivek Goyal
Quoted 12 lines · Mike Galbraith
Review
Quoted 8 lines
(quoted diff) +5
-0/* + * We may have seeky queues, don't throttle up just yet. + */ + if (time_before(jiffies, cfqd->last_seeker + CIC_SEEK_THR)) + return 0; +bzzzt. Window too large, but the though is to let them overload, but not instantly.
Quoted 3 lines
CIC_SEEK_THRis 8K jiffies so that would be 8seconds on 1000HZ system. Try using one "slice_idle" period of 8 ms. But it might turn out to be too short depending on the disk speed.
Yeah, it is too short, as is even _400_ ms. Trouble is, by the time some new task is determined to be seeky, the damage is already done.
The below does better, though not as well as "just say no to overload" of course ;-)
I have a patchlet from Corrado to test, likely better time investment than poking this darn thing with sharp sticks.
-Mike
grep elapsed testo.log
0.894345911 seconds time elapsed <== solo seeky test measurement
3.732472877 seconds time elapsed
3.208443735 seconds time elapsed
4.249776673 seconds time elapsed
2.763449260 seconds time elapsed
4.235271019 seconds time elapsed(3.73 + 3.20 + 4.24 + 2.76 + 4.23) / 5 / 0.89 = 4... darn.
block/cfq-iosched.c +30 -5
@@ -174,6 +174,8 @@ struct cfq_data {unsigned int cfq_slice_async_rq;unsigned int cfq_slice_idle; +unsigned long od_stamp; +struct list_head cic_list;/*
@@ -1296,19 +1298,26 @@ static int cfq_dispatch_requests(struct request_queue *q, int force)/* * Drain async requests before we start sync IO */ -if (cfq_cfqq_idle_window (cfqq) && cfqd->rq_in_driver[BLK_RW_ASYNC ]) +if (cfq_cfqq_idle_window (cfqq) && cfqd->rq_in_driver[BLK_RW_ASYNC ]) { + cfqd->od_stamp = jiffies;return 0 ; + }/* * If this is an async queue and we have sync IO in flight, let it wait */ -if (cfqd->sync_flight && !cfq_cfqq_sync (cfqq)) +if (cfqd->sync_flight && !cfq_cfqq_sync (cfqq)) { + cfqd->od_stamp = jiffies;return 0 ; + } max_dispatch = cfqd->cfq_quantum;if (cfq_class_idle (cfqq)) max_dispatch =1 ; +if (cfqd->busy_queues >1 ) + cfqd->od_stamp = jiffies; +/* * Does this cfqq already have too much IO in flight? */
@@ -1326,6 +1335,12 @@ static int cfq_dispatch_requests(struct request_queue *q, int force)return 0 ;/* +* Don't start overloading until we've been alone for a bit. +*/ +if (time_before (jiffies, cfqd->od_stamp + cfq_slice_sync)) +return 0 ; + +/* * we are the only queue, allow up to 4 times of 'quantum' */ if (cfqq->dispatched >=4 * max_dispatch)
@@ -1941,7 +1956,7 @@ static void cfq_update_idle_window(struct cfq_data *cfqd, struct cfq_queue *cfqq, struct cfq_io_context *cic) { - int old_idle, enable_idle; + int old_idle, enable_idle, seeky = 0;/* * Don't idle for async or idle io prio class
@@ -1949,10 +1964,19 @@ cfq_update_idle_window(struct cfq_data *cfqd, struct cfq_queue *cfqq,if (!cfq_cfqq_sync (cfqq) ||cfq_class_idle (cfqq))return ; +if (cfqd->hw_tag) { +if (CIC_SEEKY (cic)) + seeky =1 ; +/* +* If known or incalculable seekiness, delay. +*/ +if (seeky || !sample_valid (cic->seek_samples)) + cfqd->od_stamp = jiffies; + } + enable_idle = old_idle =cfq_cfqq_idle_window (cfqq); -if (!atomic_read (&cic->ioc->nr_tasks) || !cfqd->cfq_slice_idle || - (cfqd->hw_tag &&CIC_SEEKY (cic))) +if (!atomic_read (&cic->ioc->nr_tasks) || !cfqd->cfq_slice_idle || seeky) enable_idle =0 ;else if (sample_valid (cic->ttime_samples)) {if (cic->ttime_mean > cfqd->cfq_slice_idle)
@@ -2482,6 +2506,7 @@ static void *cfq_init_queue(struct request_queue *q) cfqd->cfq_slice_async_rq = cfq_slice_async_rq; cfqd->cfq_slice_idle = cfq_slice_idle; cfqd->hw_tag =1 ; + cfqd->od_stamp =INITIAL_JIFFIES ;return cfqd; }
Quoted 9 lines · Mike Galbraith
Quoted 3 lines
CIC_SEEK_THRis 8K jiffies so that would be 8seconds on 1000HZ system. Try using one "slice_idle" period of 8 ms. But it might turn out to be too short depending on the disk speed.Yeah, it is too short, as is even _400_ ms. Trouble is, by the time some new task is determined to be seeky, the damage is already done.
The below does better, though not as well as "just say no to overload" of course ;-)
So this essentially takes the "avoid impact from previous slice" to a new extreme, but idling even before dispatching requests from the new queue. We basically do two things to prevent this already - one is to only set the slice when the first request is actually serviced, and the other is to drain async requests completely before starting sync ones. I'm a bit surprised that the former doesn't solve the problem fully, I guess what happens is that if the drive has been flooded with writes, it may service the new read immediately and then return to finish emptying its writeback cache. This will cause an impact for any sync IO until that cache is flushed, and then cause that sync queue to not get as much service as it should have.
Perhaps the "set slice on first complete" isn't working correctly? Or perhaps we just need to be more extreme.
Signature · 1 line
Jens Axboe
Quoted 11 lines · Jens Axboe
Quoted 9 lines · Mike Galbraith
Quoted 3 lines
CIC_SEEK_THRis 8K jiffies so that would be 8seconds on 1000HZ system. Try using one "slice_idle" period of 8 ms. But it might turn out to be too short depending on the disk speed.Yeah, it is too short, as is even _400_ ms. Trouble is, by the time some new task is determined to be seeky, the damage is already done.
The below does better, though not as well as "just say no to overload" of course ;-)
Quoted 11 lines
So this essentially takes the "avoid impact from previous slice" to a new extreme, but idling even before dispatching requests from the new queue. We basically do two things to prevent this already - one is to only set the slice when the first request is actually serviced, and the other is to drain async requests completely before starting sync ones. I'm a bit surprised that the former doesn't solve the problem fully, I guess what happens is that if the drive has been flooded with writes, it may service the new read immediately and then return to finish emptying its writeback cache. This will cause an impact for any sync IO until that cache is flushed, and then cause that sync queue to not get as much service as it should have.
I did the stamping selection other than how long have we been solo based on these possibly wrong speculations:
If we're in the idle window and doing the async drain thing, we've at the spot where Vivek's patch helps a ton. Seemed like a great time to limit the size of any io that may land in front of my sync reader to plain "you are not alone" quantity.
If we've got sync io in flight, that should mean that my new or old known seeky queue has been serviced at least once. There's likely to be more on the way, so delay overloading then too.
The seeky bit is supposed to be the earlier "last time we saw a seeker" thing, but known seeky is too late to help a new task at all unless you turn off the overloading for ages, so I added the if incalculable check for good measure, hoping that meant the task is new, may want to exec.
Stamping any place may (see below) possibly limit the size of the io the reader can generate as well as writer, but I figured what's good for the goose is good for the the gander, or it ain't really good. The overload was causing the observed pain, definitely ain't good for both at these times at least, so don't let it do that.
Quoted 2 lines
Perhaps the "set slice on first complete" isn't working correctly? Or perhaps we just need to be more extreme.
Dunno, I was just tossing rocks and sticks at it.
I don't really understand the reasoning behind overloading: I can see that allows cutting thicker slabs for the disk, but with the streaming writer vs reader case, seems only the writers can do that. The reader is unlikely to be alone isn't it? Seems to me that either dd, a flusher thread or kjournald is going to be there with it, which gives dd a huge advantage.. it has two proxies to help it squabble over disk, konsole has none.
-Mike
Quoted 24 lines · Mike Galbraith
Quoted 11 lines · Jens Axboe
Quoted 9 lines · Mike Galbraith
Quoted 3 lines
CIC_SEEK_THRis 8K jiffies so that would be 8seconds on 1000HZ system. Try using one "slice_idle" period of 8 ms. But it might turn out to be too short depending on the disk speed.Yeah, it is too short, as is even _400_ ms. Trouble is, by the time some new task is determined to be seeky, the damage is already done.
The below does better, though not as well as "just say no to overload" of course ;-)
Quoted 11 lines
So this essentially takes the "avoid impact from previous slice" to a new extreme, but idling even before dispatching requests from the new queue. We basically do two things to prevent this already - one is to only set the slice when the first request is actually serviced, and the other is to drain async requests completely before starting sync ones. I'm a bit surprised that the former doesn't solve the problem fully, I guess what happens is that if the drive has been flooded with writes, it may service the new read immediately and then return to finish emptying its writeback cache. This will cause an impact for any sync IO until that cache is flushed, and then cause that sync queue to not get as much service as it should have.
Quoted 7 lines
I did the stamping selection other than how long have we been solo based on these possibly wrong speculations:
If we're in the idle window and doing the async drain thing, we've at the spot where Vivek's patch helps a ton. Seemed like a great time to limit the size of any io that may land in front of my sync reader to plain "you are not alone" quantity.
You can't be in the idle window and doing async drain at the same time, the idle window doesn't start until the sync queue has completed a request. Hence my above rant on device interference.
Quoted 20 lines
If we've got sync io in flight, that should mean that my new or old known seeky queue has been serviced at least once. There's likely to be more on the way, so delay overloading then too.
The seeky bit is supposed to be the earlier "last time we saw a seeker" thing, but known seeky is too late to help a new task at all unless you turn off the overloading for ages, so I added the if incalculable check for good measure, hoping that meant the task is new, may want to exec.
Stamping any place may (see below) possibly limit the size of the io the reader can generate as well as writer, but I figured what's good for the goose is good for the the gander, or it ain't really good. The overload was causing the observed pain, definitely ain't good for both at these times at least, so don't let it do that.
Quoted 2 lines
Perhaps the "set slice on first complete" isn't working correctly? Or perhaps we just need to be more extreme.
Dunno, I was just tossing rocks and sticks at it.
Quoted 7 lines
I don't really understand the reasoning behind overloading: I can see that allows cutting thicker slabs for the disk, but with the streaming writer vs reader case, seems only the writers can do that. The reader is unlikely to be alone isn't it? Seems to me that either dd, a flusher thread or kjournald is going to be there with it, which gives dd a huge advantage.. it has two proxies to help it squabble over disk, konsole has none.
That is true, async queues have a huge advantage over sync ones. But sync vs async is only part of it, any combination of queued sync, queued sync random etc have different ramifications on behaviour of the individual queue.
It's not hard to make the latency good, the hard bit is making sure we also perform well for all other scenarios.
Signature · 1 line
Jens Axboe
Quoted 10 lines · Jens Axboe
Quoted 4 lines · Mike Galbraith
If we're in the idle window and doing the async drain thing, we've at the spot where Vivek's patch helps a ton. Seemed like a great time to limit the size of any io that may land in front of my sync reader to plain "you are not alone" quantity.
You can't be in the idle window and doing async drain at the same time, the idle window doesn't start until the sync queue has completed a request. Hence my above rant on device interference.
I'll take your word for it.
/*
* Drain async requests before we start sync IO
*/
if (cfq_cfqq_idle_window(cfqq) && cfqd->rq_in_driver[BLK_RW_ASYNC])Looked about the same to me as..
enable_idle = old_idle = cfq_cfqq_idle_window(cfqq);
..where Vivek prevented turning 1 into 0, so I stamped it ;-)
Quoted 10 lines
Quoted 9 lines
Dunno, I was just tossing rocks and sticks at it.
I don't really understand the reasoning behind overloading: I can see that allows cutting thicker slabs for the disk, but with the streaming writer vs reader case, seems only the writers can do that. The reader is unlikely to be alone isn't it? Seems to me that either dd, a flusher thread or kjournald is going to be there with it, which gives dd a huge advantage.. it has two proxies to help it squabble over disk, konsole has none.
Quoted 7 lines
That is true, async queues have a huge advantage over sync ones. But sync vs async is only part of it, any combination of queued sync, queued sync random etc have different ramifications on behaviour of the individual queue.
It's not hard to make the latency good, the hard bit is making sure we also perform well for all other scenarios.
Yeah, that's why I'm trying to be careful about what I say, I know full well this ain't easy to get right. I'm not even thinking of submitting anything, it's just diagnostic testing.
WRT my who can overload theory, I instrumented for my own edification.
Overload totally forbidden, stamps ergo disabled.
fairness=0 11.3 avg (ie == virgin source)
fairness=1 2.8 avg
Back to virgin settings, instrument who is overloading during sequences of..
echo 2 > /proc/sys/vm/drop_caches
sh -c "perf stat -- konsole -e exit" 2>&1|tee -a $LOGFILE
..with dd continually running.
1 second counts for above.
...
[ 916.585880] od_sync: 0 od_async: 87 reject_sync: 0 reject_async: 37 [ 917.662585] od_sync: 0 od_async: 126 reject_sync: 0 reject_async: 53 [ 918.732872] od_sync: 0 od_async: 96 reject_sync: 0 reject_async: 22 [ 919.743730] od_sync: 0 od_async: 75 reject_sync: 0 reject_async: 15 [ 920.914549] od_sync: 0 od_async: 81 reject_sync: 0 reject_async: 17 [ 921.988198] od_sync: 0 od_async: 123 reject_sync: 0 reject_async: 30
...minutes long
(reject == fqq->dispatched >= 4 * max_dispatch)
Doing the same with firefox, I did see the burst below one time, dunno what triggered that. I watched 6 runs, and only saw such a burst once. Typically, numbers are the same as konsole, with a very rare 4 or 5 for sync sneaking in.
[ 1988.177758] od_sync: 0 od_async: 104 reject_sync: 0 reject_async: 48 [ 1992.291779] od_sync: 19 od_async: 83 reject_sync: 0 reject_async: 82 [ 1993.300850] od_sync: 79 od_async: 0 reject_sync: 28 reject_async: 0 [ 1994.313327] od_sync: 147 od_async: 104 reject_sync: 90 reject_async: 16 [ 1995.378025] od_sync: 14 od_async: 45 reject_sync: 0 reject_async: 2 [ 1996.456871] od_sync: 15 od_async: 74 reject_sync: 1 reject_async: 7 [ 1997.611226] od_sync: 0 od_async: 84 reject_sync: 0 reject_async: 14
Never noticed a sync overload watching a make -j4 for a couple minutes.
Quoted 7 lines
WRT my who can overload theory, I instrumented for my own edification.
Overload totally forbidden, stamps ergo disabled.
fairness=0 11.3 avg (ie == virgin source)
fairness=1 2.8 avg
(oops, quantum was set to 16 as well there. not that it matters, but for completeness)
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 7 lines
WRT my who can overload theory, I instrumented for my own edification.
Overload totally forbidden, stamps ergo disabled.
fairness=0 11.3 avg (ie == virgin source)
fairness=1 2.8 avg
(oops, quantum was set to 16 as well there. not that it matters, but for completeness)
Quoted 19 lines · Mike Galbraith
Quoted 10 lines · Jens Axboe
Quoted 4 lines · Mike Galbraith
If we're in the idle window and doing the async drain thing, we've at the spot where Vivek's patch helps a ton. Seemed like a great time to limit the size of any io that may land in front of my sync reader to plain "you are not alone" quantity.
You can't be in the idle window and doing async drain at the same time, the idle window doesn't start until the sync queue has completed a request. Hence my above rant on device interference.
I'll take your word for it.
/* * Drain async requests before we start sync IO */ if (cfq_cfqq_idle_window(cfqq) && cfqd->rq_in_driver[BLK_RW_ASYNC])
Quoted 5 lines
Looked about the same to me as..
enable_idle = old_idle = cfq_cfqq_idle_window(cfqq);..where Vivek prevented turning 1 into 0, so I stamped it ;-)
cfq_cfqq_idle_window(cfqq) just tells you whether this queue may enter idling, not that it is currently idling. The actual idling happens from cfq_completed_request(), here:
else if (cfqq_empty && !cfq_close_cooperator(cfqd, cfqq, 1) &&
sync && !rq_noidle(rq))
cfq_arm_slice_timer(cfqd);and after that the queue will be marked as waiting, so cfq_cfqq_wait_request(cfqq) is a better indication of whether we are currently waiting for a request (idling) or not.
Quoted 15 lines
Quoted 15 lines
Quoted 9 lines
Dunno, I was just tossing rocks and sticks at it.
I don't really understand the reasoning behind overloading: I can see that allows cutting thicker slabs for the disk, but with the streaming writer vs reader case, seems only the writers can do that. The reader is unlikely to be alone isn't it? Seems to me that either dd, a flusher thread or kjournald is going to be there with it, which gives dd a huge advantage.. it has two proxies to help it squabble over disk, konsole has none.
That is true, async queues have a huge advantage over sync ones. But sync vs async is only part of it, any combination of queued sync, queued sync random etc have different ramifications on behaviour of the individual queue.
Quoted 6 lines
Quoted 2 lines
It's not hard to make the latency good, the hard bit is making sure we also perform well for all other scenarios.
Yeah, that's why I'm trying to be careful about what I say, I know full well this ain't easy to get right. I'm not even thinking of submitting anything, it's just diagnostic testing.
It's much appreciated btw, if we can make this better without killing throughput, then I'm surely interested in picking up your interesting bits and getting them massaged into something we can include. So don't be discouraged, I'm just being realistic :-)
Signature · 1 line
Jens Axboe
Quoted 19 lines · Mike Galbraith
Quoted 10 lines · Jens Axboe
Quoted 4 lines · Mike Galbraith
If we're in the idle window and doing the async drain thing, we've at the spot where Vivek's patch helps a ton. Seemed like a great time to limit the size of any io that may land in front of my sync reader to plain "you are not alone" quantity.
You can't be in the idle window and doing async drain at the same time, the idle window doesn't start until the sync queue has completed a request. Hence my above rant on device interference.
I'll take your word for it.
/* * Drain async requests before we start sync IO */ if (cfq_cfqq_idle_window(cfqq) && cfqd->rq_in_driver[BLK_RW_ASYNC])
Quoted 5 lines
Looked about the same to me as..
enable_idle = old_idle = cfq_cfqq_idle_window(cfqq);..where Vivek prevented turning 1 into 0, so I stamped it ;-)
cfq_cfqq_idle_window(cfqq) just tells you whether this queue may enter idling, not that it is currently idling. The actual idling happens from cfq_completed_request(), here:
else if (cfqq_empty && !cfq_close_cooperator(cfqd, cfqq, 1) &&
sync && !rq_noidle(rq))
cfq_arm_slice_timer(cfqd);and after that the queue will be marked as waiting, so cfq_cfqq_wait_request(cfqq) is a better indication of whether we are currently waiting for a request (idling) or not.
Quoted 15 lines
Quoted 15 lines
Quoted 9 lines
Dunno, I was just tossing rocks and sticks at it.
I don't really understand the reasoning behind overloading: I can see that allows cutting thicker slabs for the disk, but with the streaming writer vs reader case, seems only the writers can do that. The reader is unlikely to be alone isn't it? Seems to me that either dd, a flusher thread or kjournald is going to be there with it, which gives dd a huge advantage.. it has two proxies to help it squabble over disk, konsole has none.
That is true, async queues have a huge advantage over sync ones. But sync vs async is only part of it, any combination of queued sync, queued sync random etc have different ramifications on behaviour of the individual queue.
Quoted 6 lines
Quoted 2 lines
It's not hard to make the latency good, the hard bit is making sure we also perform well for all other scenarios.
Yeah, that's why I'm trying to be careful about what I say, I know full well this ain't easy to get right. I'm not even thinking of submitting anything, it's just diagnostic testing.
It's much appreciated btw, if we can make this better without killing throughput, then I'm surely interested in picking up your interesting bits and getting them massaged into something we can include. So don't be discouraged, I'm just being realistic :-)
Signature · 1 line
Jens Axboe
Quoted 17 lines · Jens Axboe
Quoted 11 lines · Mike Galbraith
/* * Drain async requests before we start sync IO */ if (cfq_cfqq_idle_window(cfqq) && cfqd->rq_in_driver[BLK_RW_ASYNC])Looked about the same to me as..
enable_idle = old_idle = cfq_cfqq_idle_window(cfqq);..where Vivek prevented turning 1 into 0, so I stamped it ;-)
cfq_cfqq_idle_window(cfqq)just tells you whether this queue may enter idling, not that it is currently idling. The actual idling happens fromcfq_completed_request(), here:
Quoted 7 lines
else if (cfqq_empty && !cfq_close_cooperator(cfqd, cfqq, 1) && sync && !rq_noidle(rq)) cfq_arm_slice_timer(cfqd);and after that the queue will be marked as waiting, so
cfq_cfqq_wait_request(cfqq)is a better indication of whether we are currently waiting for a request (idling) or not.
Hm. Then cfq_cfqq_idle_window(cfqq) actually suits my intent better.
(If I want to reduce async's advantage, I should target specifically, ie only stamp if this queue is a sync queue....otoh, if this queue is sync, it is now officially too late, whereas if this queue is dd about to inflict the wrath of kjournald on my reader's world, stamping now is a really good idea.. scritch scritch scritch <smoke>)
I'll go tinker with it. Thanks for the clue.
-Mike
A message that is not in the archive (sent off-list or not yet received): 20091002095555.GB26962-tSWWG44O7X1aa/[email protected]
Quoted 17 lines · Jens Axboe
Quoted 11 lines · Mike Galbraith
/* * Drain async requests before we start sync IO */ if (cfq_cfqq_idle_window(cfqq) && cfqd->rq_in_driver[BLK_RW_ASYNC])Looked about the same to me as..
enable_idle = old_idle = cfq_cfqq_idle_window(cfqq);..where Vivek prevented turning 1 into 0, so I stamped it ;-)
cfq_cfqq_idle_window(cfqq)just tells you whether this queue may enter idling, not that it is currently idling. The actual idling happens fromcfq_completed_request(), here:
Quoted 7 lines
else if (cfqq_empty && !cfq_close_cooperator(cfqd, cfqq, 1) && sync && !rq_noidle(rq)) cfq_arm_slice_timer(cfqd);and after that the queue will be marked as waiting, so
cfq_cfqq_wait_request(cfqq)is a better indication of whether we are currently waiting for a request (idling) or not.
Hm. Then cfq_cfqq_idle_window(cfqq) actually suits my intent better.
(If I want to reduce async's advantage, I should target specifically, ie only stamp if this queue is a sync queue....otoh, if this queue is sync, it is now officially too late, whereas if this queue is dd about to inflict the wrath of kjournald on my reader's world, stamping now is a really good idea.. scritch scritch scritch <smoke>)
I'll go tinker with it. Thanks for the clue.
-Mike
A message that is not in the archive (sent off-list or not yet received): 20091002080417.GG14918-tSWWG44O7X1aa/[email protected]
Quoted 10 lines · Jens Axboe
Quoted 4 lines · Mike Galbraith
If we're in the idle window and doing the async drain thing, we've at the spot where Vivek's patch helps a ton. Seemed like a great time to limit the size of any io that may land in front of my sync reader to plain "you are not alone" quantity.
You can't be in the idle window and doing async drain at the same time, the idle window doesn't start until the sync queue has completed a request. Hence my above rant on device interference.
I'll take your word for it.
/*
* Drain async requests before we start sync IO
*/
if (cfq_cfqq_idle_window(cfqq) && cfqd->rq_in_driver[BLK_RW_ASYNC])Looked about the same to me as..
enable_idle = old_idle = cfq_cfqq_idle_window(cfqq);
..where Vivek prevented turning 1 into 0, so I stamped it ;-)
Quoted 10 lines
Quoted 9 lines
Dunno, I was just tossing rocks and sticks at it.
I don't really understand the reasoning behind overloading: I can see that allows cutting thicker slabs for the disk, but with the streaming writer vs reader case, seems only the writers can do that. The reader is unlikely to be alone isn't it? Seems to me that either dd, a flusher thread or kjournald is going to be there with it, which gives dd a huge advantage.. it has two proxies to help it squabble over disk, konsole has none.
Quoted 7 lines
That is true, async queues have a huge advantage over sync ones. But sync vs async is only part of it, any combination of queued sync, queued sync random etc have different ramifications on behaviour of the individual queue.
It's not hard to make the latency good, the hard bit is making sure we also perform well for all other scenarios.
Yeah, that's why I'm trying to be careful about what I say, I know full well this ain't easy to get right. I'm not even thinking of submitting anything, it's just diagnostic testing.
WRT my who can overload theory, I instrumented for my own edification.
Overload totally forbidden, stamps ergo disabled.
fairness=0 11.3 avg (ie == virgin source)
fairness=1 2.8 avg
Back to virgin settings, instrument who is overloading during sequences of..
echo 2 > /proc/sys/vm/drop_caches
sh -c "perf stat -- konsole -e exit" 2>&1|tee -a $LOGFILE
..with dd continually running.
1 second counts for above.
...
[ 916.585880] od_sync: 0 od_async: 87 reject_sync: 0 reject_async: 37 [ 917.662585] od_sync: 0 od_async: 126 reject_sync: 0 reject_async: 53 [ 918.732872] od_sync: 0 od_async: 96 reject_sync: 0 reject_async: 22 [ 919.743730] od_sync: 0 od_async: 75 reject_sync: 0 reject_async: 15 [ 920.914549] od_sync: 0 od_async: 81 reject_sync: 0 reject_async: 17 [ 921.988198] od_sync: 0 od_async: 123 reject_sync: 0 reject_async: 30
...minutes long
(reject == fqq->dispatched >= 4 * max_dispatch)
Doing the same with firefox, I did see the burst below one time, dunno what triggered that. I watched 6 runs, and only saw such a burst once. Typically, numbers are the same as konsole, with a very rare 4 or 5 for sync sneaking in.
[ 1988.177758] od_sync: 0 od_async: 104 reject_sync: 0 reject_async: 48 [ 1992.291779] od_sync: 19 od_async: 83 reject_sync: 0 reject_async: 82 [ 1993.300850] od_sync: 79 od_async: 0 reject_sync: 28 reject_async: 0 [ 1994.313327] od_sync: 147 od_async: 104 reject_sync: 90 reject_async: 16 [ 1995.378025] od_sync: 14 od_async: 45 reject_sync: 0 reject_async: 2 [ 1996.456871] od_sync: 15 od_async: 74 reject_sync: 1 reject_async: 7 [ 1997.611226] od_sync: 0 od_async: 84 reject_sync: 0 reject_async: 14
Never noticed a sync overload watching a make -j4 for a couple minutes.
Quoted 2 lines · Jens Axboe
It's not hard to make the latency good, the hard bit is making sure we also perform well for all other scenarios.
Looking at the numbers from Mike:
| dd competing against perf stat -- konsole -e exec timings, 5 back to | back runs | Avg | before 9.15 14.51 9.39 15.06 9.90 11.6 | after [+patch] 1.76 1.54 1.93 1.88 1.56 1.7
_PLEASE_ make read latencies this good - the numbers are _vastly_ better. We'll worry about the 'other' things _after_ we've reached good latencies.
I thought this principle was a well established basic rule of Linux IO scheduling. Why do we have to have a 'latency vs. bandwidth' discussion again and again? I thought latency won hands down.
Ingo
Quoted 2 lines · Jens Axboe
It's not hard to make the latency good, the hard bit is making sure we also perform well for all other scenarios.
Looking at the numbers from Mike:
| dd competing against perf stat -- konsole -e exec timings, 5 back to | back runs | Avg | before 9.15 14.51 9.39 15.06 9.90 11.6 | after [+patch] 1.76 1.54 1.93 1.88 1.56 1.7
_PLEASE_ make read latencies this good - the numbers are _vastly_ better. We'll worry about the 'other' things _after_ we've reached good latencies.
I thought this principle was a well established basic rule of Linux IO scheduling. Why do we have to have a 'latency vs. bandwidth' discussion again and again? I thought latency won hands down.
Ingo
Quoted 14 lines · Ingo Molnar
Quoted 2 lines · Jens Axboe
It's not hard to make the latency good, the hard bit is making sure we also perform well for all other scenarios.
Looking at the numbers from Mike:
| dd competing against perf stat -- konsole -e exec timings, 5 back to | back runs | Avg | before 9.15 14.51 9.39 15.06 9.90 11.6 | after [+patch] 1.76 1.54 1.93 1.88 1.56 1.7
Quoted 7 lines
_PLEASE_ make read latencies this good - the numbers are _vastly_ better. We'll worry about the 'other' things _after_ we've reached good latencies.
I thought this principle was a well established basic rule of Linux IO scheduling. Why do we have to have a 'latency vs. bandwidth' discussion again and again? I thought latency won hands down.
It's really not that simple, if we go and do easy latency bits, then throughput drops 30% or more. You can't say it's black and white latency vs throughput issue, that's just not how the real world works. The server folks would be most unpleased.
Signature · 1 line
Jens Axboe
Quoted 3 lines · Jens Axboe
It's really not that simple, if we go and do easy latency bits, then throughput drops 30% or more.
Well, if we're talking 500-950% improvement vs 30% deprovement, I think it's pretty clear, though. Even the server people do care about latencies.
Often they care quite a bit, in fact.
And Mike's patch didn't look big or complicated.
Quoted 1 line
You can't say it's black and white latency vs throughput issue,
Umm. Almost 1000% vs 30%. Forget latency vs throughput. That's pretty damn black-and-white _regardless_ of what you're measuring. Plus you probably made up the 30% - have you tested the patch?
And quite frankly, we get a _lot_ of complaints about latency. A LOT. It's just harder to measure, so people seldom attach numbers to it. But that again means that when people _are_ able to attach numbers to it, we should take those numbers _more_ seriously rather than less.
So the 30% you threw out as a number is pretty much worthless.
Linus
Quoted 6 lines · Linus Torvalds
Quoted 3 lines · Jens Axboe
It's really not that simple, if we go and do easy latency bits, then throughput drops 30% or more.
Quoted 6 lines
Well, if we're talking 500-950% improvement vs 30% deprovement, I think it's pretty clear, though. Even the server people do care about latencies.
Often they care quite a bit, in fact.
And Mike's patch didn't look big or complicated.
But it is a hack. (thought about and measured, but hack nonetheless)
I haven't tested it on much other than reader vs streaming writer. It may well destroy the rest of the IO universe. I don't have the hw to even test any hairy chested IO.
-Mike
Quoted 12 lines · Mike Galbraith
Quoted 11 lines · Linus Torvalds
Quoted 3 lines · Jens Axboe
It's really not that simple, if we go and do easy latency bits, then throughput drops 30% or more.
Well, if we're talking 500-950% improvement vs 30% deprovement, I think it's pretty clear, though. Even the server people do care about latencies.
Often they care quite a bit, in fact.
Quoted 7 lines
Quoted 1 line
And Mike's patch didn't look big or complicated.
But it is a hack. (thought about and measured, but hack nonetheless)
I haven't tested it on much other than reader vs streaming writer. It may well destroy the rest of the IO universe. I don't have the hw to even test any hairy chested IO.
I'll get a desktop box going on this too. The plan is to make the latency as good as we can without making too many stupid decisions in the io scheduler, then we can care about the throughput later. Rinse and repeat.
Signature · 1 line
Jens Axboe
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 12 lines · Mike Galbraith
Quoted 11 lines · Linus Torvalds
Quoted 3 lines · Jens Axboe
It's really not that simple, if we go and do easy latency bits, then throughput drops 30% or more.
Well, if we're talking 500-950% improvement vs 30% deprovement, I think it's pretty clear, though. Even the server people do care about latencies.
Often they care quite a bit, in fact.
Quoted 7 lines
Quoted 1 line
And Mike's patch didn't look big or complicated.
But it is a hack. (thought about and measured, but hack nonetheless)
I haven't tested it on much other than reader vs streaming writer. It may well destroy the rest of the IO universe. I don't have the hw to even test any hairy chested IO.
I'll get a desktop box going on this too. The plan is to make the latency as good as we can without making too many stupid decisions in the io scheduler, then we can care about the throughput later. Rinse and repeat.
Signature · 1 line
Jens Axboe
A message that is not in the archive (sent off-list or not yet received): alpine.LFD.2.01.0910020715160.6996-bi+AKbBUZKY6gyzm1THtWbp2dZbC/[email protected]
Quoted 6 lines · Linus Torvalds
Quoted 3 lines · Jens Axboe
It's really not that simple, if we go and do easy latency bits, then throughput drops 30% or more.
Quoted 6 lines
Well, if we're talking 500-950% improvement vs 30% deprovement, I think it's pretty clear, though. Even the server people do care about latencies.
Often they care quite a bit, in fact.
And Mike's patch didn't look big or complicated.
But it is a hack. (thought about and measured, but hack nonetheless)
I haven't tested it on much other than reader vs streaming writer. It may well destroy the rest of the IO universe. I don't have the hw to even test any hairy chested IO.
-Mike
Quoted 11 lines · Linus Torvalds
Quoted 3 lines · Jens Axboe
It's really not that simple, if we go and do easy latency bits, then throughput drops 30% or more.
Well, if we're talking 500-950% improvement vs 30% deprovement, I think it's pretty clear, though. Even the server people do care about latencies.
Often they care quite a bit, in fact.
Mostly they care about throughput, and when they come running because some their favorite app/benchmark/etc is now 2% slower, I get to hear about it all the time. So yes, latency is not ignored, but mostly they yack about throughput.
Quoted 1 line
And Mike's patch didn't look big or complicated.
It wasn't, it was more of a hack than something mergeable though (and I think Mike will agree on that). So I'll repeat what I said to Mike, I'm very well prepared to get something worked out and merged and I very much appreciate the work he's putting into this.
Quoted 5 lines
Quoted 1 line
You can't say it's black and white latency vs throughput issue,
Umm. Almost 1000% vs 30%. Forget latency vs throughput. That's pretty damn black-and-white _regardless_ of what you're measuring. Plus you probably made up the 30% - have you tested the patch?
The 30% is totally made up, it's based on previous latency vs throughput tradeoffs. I haven't tested Mike's patch.
Quoted 4 lines
And quite frankly, we get a _lot_ of complaints about latency. A LOT. It's just harder to measure, so people seldom attach numbers to it. But that again means that when people _are_ able to attach numbers to it, we should take those numbers _more_ seriously rather than less.
I agree, we can easily make CFQ be very about about latency. If you think that is fine, then lets just do that. Then we'll get to fix the server side up when the next RHEL/SLES/whatever cycle is honing in on a kernel, hopefully we wont have to start over when that happens.
Quoted 1 line
So the 30% you threw out as a number is pretty much worthless.
It's hand waving, definitely. But I've been doing io scheduler tweaking for years, and I know how hard it is to balance. If you want latency, then you basically only ever give the device 1 thing to do. And you let things cool down before switching over. If you do that, then your nice big array of SSDs or rotating drives will easily drop to 1/4th of the original performance. So we try and tweak the logic to make everybody happy.
In some cases I wish we had a server vs desktop switch, since it would decisions on this easier. I know you say that servers care about latency, but not at all to the extent that desktops do. Most desktop users would gladly give away the top of the performance for latency, that's not true of most server users. Depends on what the server does, of course.
Signature · 1 line
Jens Axboe
Quoted 10 lines · Linus Torvalds
Quoted 3 lines · Jens Axboe
It's really not that simple, if we go and do easy latency bits, then throughput drops 30% or more.
Well, if we're talking 500-950% improvement vs 30% deprovement, I think it's pretty clear, though. Even the server people do care about latencies.
Often they care quite a bit, in fact.
The other thing is that latency is basically a given property in any system - as an app writer you have to live with it, there's not much you can do to improve it.
Bandwidth on the other hand is a lot more engineerable, as it tends to be about batching things and you can batch in user-space too. Batching is often easier to do than getting good latencies.
Then there's also the fact that the range of apps that care about bandwidth is a lot smaller than the range of apps which care about latencies. The default should help more apps - i.e. latencies.
Ingo
Quoted 11 lines · Linus Torvalds
Quoted 3 lines · Jens Axboe
It's really not that simple, if we go and do easy latency bits, then throughput drops 30% or more.
Well, if we're talking 500-950% improvement vs 30% deprovement, I think it's pretty clear, though. Even the server people do care about latencies.
Often they care quite a bit, in fact.
Mostly they care about throughput, and when they come running because some their favorite app/benchmark/etc is now 2% slower, I get to hear about it all the time. So yes, latency is not ignored, but mostly they yack about throughput.
Quoted 1 line
And Mike's patch didn't look big or complicated.
It wasn't, it was more of a hack than something mergeable though (and I think Mike will agree on that). So I'll repeat what I said to Mike, I'm very well prepared to get something worked out and merged and I very much appreciate the work he's putting into this.
Quoted 5 lines
Quoted 1 line
You can't say it's black and white latency vs throughput issue,
Umm. Almost 1000% vs 30%. Forget latency vs throughput. That's pretty damn black-and-white _regardless_ of what you're measuring. Plus you probably made up the 30% - have you tested the patch?
The 30% is totally made up, it's based on previous latency vs throughput tradeoffs. I haven't tested Mike's patch.
Quoted 4 lines
And quite frankly, we get a _lot_ of complaints about latency. A LOT. It's just harder to measure, so people seldom attach numbers to it. But that again means that when people _are_ able to attach numbers to it, we should take those numbers _more_ seriously rather than less.
I agree, we can easily make CFQ be very about about latency. If you think that is fine, then lets just do that. Then we'll get to fix the server side up when the next RHEL/SLES/whatever cycle is honing in on a kernel, hopefully we wont have to start over when that happens.
Quoted 1 line
So the 30% you threw out as a number is pretty much worthless.
It's hand waving, definitely. But I've been doing io scheduler tweaking for years, and I know how hard it is to balance. If you want latency, then you basically only ever give the device 1 thing to do. And you let things cool down before switching over. If you do that, then your nice big array of SSDs or rotating drives will easily drop to 1/4th of the original performance. So we try and tweak the logic to make everybody happy.
In some cases I wish we had a server vs desktop switch, since it would decisions on this easier. I know you say that servers care about latency, but not at all to the extent that desktops do. Most desktop users would gladly give away the top of the performance for latency, that's not true of most server users. Depends on what the server does, of course.
Signature · 1 line
Jens Axboe
Quoted 5 lines · Jens Axboe
Mostly they care about throughput, and when they come running because some their favorite app/benchmark/etc is now 2% slower, I get to hear about it all the time. So yes, latency is not ignored, but mostly they yack about throughput.
The reason they yack about it is that they can measure it.
Give them the benchmark where it goes the other way, and tell them why they see a 2% deprovement. Give them some button they can tweak, because they will.
But make the default be low-latency. Because everybody cares about low latency, and the people who do so are _not_ the people who you give buttons to tweak things with.
Quoted 4 lines
I agree, we can easily make CFQ be very about about latency. If you think that is fine, then lets just do that. Then we'll get to fix the server side up when the next RHEL/SLES/whatever cycle is honing in on a kernel, hopefully we wont have to start over when that happens.
I really think we should do latency first, and throughput second.
It's _easy_ to get throughput. The people who care just about throughput can always just disable all the work we do for latency. If they really care about just throughput, they won't want fairness either - none of that complex stuff.
Linus
Quoted 2 lines · Linus Torvalds
I really think we should do latency first, and throughput second.
Agree.
Quoted 1 line
It's _easy_ to get throughput. The people who care just about throughput can always just disable all the work we do for latency.
But in my experience it is not that simple...
The argument latency vs throughput or desktop vs server is wrong.
I/O can never keep up with the ability of CPUs to dirty data.
On desktops and servers (really many-user-desktops) we want minimum latency but the enemy is dirty VM. If we ignore the need for throughput to flush dirty pages, VM gets angry and forced VM page cleaning I/O is bad I/O.
We want min latency with low dirty page percent but need to switch to max write throughput at some high dirty page percent.
We can not prevent the cliff we fall off where the system chokes because the dirty page load is too high, but if we only worry about latency, we bring that choke point cliff in so it happens with a lower load. A 10% lower overload point might be fine to get 100% better latency, but would desktop users accept a 50% lower overload point where running one more application makes the system appear hung?
Even desktop users commonly measure "how much work can I do before the system becomes unresponsive".
jim
A message that is not in the archive (sent off-list or not yet received): alpine.LFD.2.01.0910020811490.6996-bi+AKbBUZKY6gyzm1THtWbp2dZbC/[email protected]
Quoted 2 lines · Linus Torvalds
I really think we should do latency first, and throughput second.
Agree.
Quoted 2 lines
It's _easy_ to get throughput. The people who care just about throughput can always just disable all the work we do for latency.
But in my experience it is not that simple...
The argument latency vs throughput or desktop vs server is wrong.
I/O can never keep up with the ability of CPUs to dirty data.
On desktops and servers (really many-user-desktops) we want minimum latency but the enemy is dirty VM. If we ignore the need for throughput to flush dirty pages, VM gets angry and forced VM page cleaning I/O is bad I/O.
We want min latency with low dirty page percent but need to switch to max write throughput at some high dirty page percent.
We can not prevent the cliff we fall off where the system chokes because the dirty page load is too high, but if we only worry about latency, we bring that choke point cliff in so it happens with a lower load. A 10% lower overload point might be fine to get 100% better latency, but would desktop users accept a 50% lower overload point where running one more application makes the system appear hung?
Even desktop users commonly measure "how much work can I do before the system becomes unresponsive".
jim
Quoted 9 lines · Linus Torvalds
Quoted 5 lines · Jens Axboe
Mostly they care about throughput, and when they come running because some their favorite app/benchmark/etc is now 2% slower, I get to hear about it all the time. So yes, latency is not ignored, but mostly they yack about throughput.
Quoted 5 lines
The reason they yack about it is that they can measure it.
Give them the benchmark where it goes the other way, and tell them why they see a 2% deprovement. Give them some button they can tweak, because they will.
To some extent that's true, and I didn't want to generalize. If they are adament that the benchmark models their real life, then no amount of pointing in the other direction will change that.
Your point about tuning is definitely true, these people are used to tuning things. For the desktop we care a lot more about working out of the box.
Quoted 3 lines
But make the default be low-latency. Because everybody cares about low latency, and the people who do so are _not_ the people who you give buttons to tweak things with.
Totally agree.
Quoted 11 lines
Quoted 4 lines
I agree, we can easily make CFQ be very about about latency. If you think that is fine, then lets just do that. Then we'll get to fix the server side up when the next RHEL/SLES/whatever cycle is honing in on a kernel, hopefully we wont have to start over when that happens.
I really think we should do latency first, and throughput second.
It's _easy_ to get throughput. The people who care just about throughput can always just disable all the work we do for latency. If they really care about just throughput, they won't want fairness either - none of that complex stuff.
It's not _that_ easy, it depends a lot on the access patterns. A good example of that is actually the idling that we already do. Say you have two applications, each starting up. If you start them both at the same time and just care for the dumb low latency, then you'll do one IO from each of them in turn. Latency will be good, but throughput will be aweful. And this means that in 20s they are both started, while with the slice idling and priority disk access that CFQ does, you'd hopefully have both up and running in 2s.
So latency is good, definitely, but sometimes you have to worry about the bigger picture too. Latency is more than single IOs, it's often for complete operation which may involve lots of IOs. Single IO latency is a benchmark thing, it's not a real life issue. And that's where it becomes complex and not so black and white. Mike's test is a really good example of that.
Signature · 1 line
Jens Axboe
Quoted 9 lines · Linus Torvalds
Quoted 5 lines · Jens Axboe
Mostly they care about throughput, and when they come running because some their favorite app/benchmark/etc is now 2% slower, I get to hear about it all the time. So yes, latency is not ignored, but mostly they yack about throughput.
Quoted 5 lines
The reason they yack about it is that they can measure it.
Give them the benchmark where it goes the other way, and tell them why they see a 2% deprovement. Give them some button they can tweak, because they will.
To some extent that's true, and I didn't want to generalize. If they are adament that the benchmark models their real life, then no amount of pointing in the other direction will change that.
Your point about tuning is definitely true, these people are used to tuning things. For the desktop we care a lot more about working out of the box.
Quoted 3 lines
But make the default be low-latency. Because everybody cares about low latency, and the people who do so are _not_ the people who you give buttons to tweak things with.
Totally agree.
Quoted 11 lines
Quoted 4 lines
I agree, we can easily make CFQ be very about about latency. If you think that is fine, then lets just do that. Then we'll get to fix the server side up when the next RHEL/SLES/whatever cycle is honing in on a kernel, hopefully we wont have to start over when that happens.
I really think we should do latency first, and throughput second.
It's _easy_ to get throughput. The people who care just about throughput can always just disable all the work we do for latency. If they really care about just throughput, they won't want fairness either - none of that complex stuff.
It's not _that_ easy, it depends a lot on the access patterns. A good example of that is actually the idling that we already do. Say you have two applications, each starting up. If you start them both at the same time and just care for the dumb low latency, then you'll do one IO from each of them in turn. Latency will be good, but throughput will be aweful. And this means that in 20s they are both started, while with the slice idling and priority disk access that CFQ does, you'd hopefully have both up and running in 2s.
So latency is good, definitely, but sometimes you have to worry about the bigger picture too. Latency is more than single IOs, it's often for complete operation which may involve lots of IOs. Single IO latency is a benchmark thing, it's not a real life issue. And that's where it becomes complex and not so black and white. Mike's test is a really good example of that.
Signature · 1 line
Jens Axboe
Quoted 9 lines · Jens Axboe
It's not _that_ easy, it depends a lot on the access patterns. A good example of that is actually the idling that we already do. Say you have two applications, each starting up. If you start them both at the same time and just care for the dumb low latency, then you'll do one IO from each of them in turn. Latency will be good, but throughput will be aweful. And this means that in 20s they are both started, while with the slice idling and priority disk access that CFQ does, you'd hopefully have both up and running in 2s.
Quoted 6 lines
So latency is good, definitely, but sometimes you have to worry about the bigger picture too. Latency is more than single IOs, it's often for complete operation which may involve lots of IOs. Single IO latency is a benchmark thing, it's not a real life issue. And that's where it becomes complex and not so black and white. Mike's test is a really good example of that.
To the extent of you arguing that Mike's test is artificial (i'm not sure you are arguing that) - Mike certainly did not do an artificial test - he tested 'konsole' cache-cold startup latency, such as:
sh -c "perf stat -- konsole -e exit" 2>&1|tee -a $LOGFILE
against a streaming dd.
That is a _very_ relevant benchmark IMHO and konsole's cache footprint is far from trivial. (In fact i'd argue it's one of the most important IO benchmarks on a desktop system - how does your desktop hold up to something doing streaming IO.)
Ingo
Quoted 19 lines · Ingo Molnar
Quoted 9 lines · Jens Axboe
It's not _that_ easy, it depends a lot on the access patterns. A good example of that is actually the idling that we already do. Say you have two applications, each starting up. If you start them both at the same time and just care for the dumb low latency, then you'll do one IO from each of them in turn. Latency will be good, but throughput will be aweful. And this means that in 20s they are both started, while with the slice idling and priority disk access that CFQ does, you'd hopefully have both up and running in 2s.
Quoted 6 lines
So latency is good, definitely, but sometimes you have to worry about the bigger picture too. Latency is more than single IOs, it's often for complete operation which may involve lots of IOs. Single IO latency is a benchmark thing, it's not a real life issue. And that's where it becomes complex and not so black and white. Mike's test is a really good example of that.
Quoted 3 lines
To the extent of you arguing that Mike's test is artificial (i'm not sure you are arguing that) - Mike certainly did not do an artificial test - he tested 'konsole' cache-cold startup latency, such as:
[snip]
I was saying the exact opposite, that Mike's test is a good example of a valid test. It's not measuring single IO latencies, it's doing a sequence of valid events and looking at the latency for those. It's benchmarking the bigger picture, not a microbenchmark.
Signature · 1 line
Jens Axboe
Quoted 24 lines · Jens Axboe
Quoted 19 lines · Ingo Molnar
Quoted 9 lines · Jens Axboe
It's not _that_ easy, it depends a lot on the access patterns. A good example of that is actually the idling that we already do. Say you have two applications, each starting up. If you start them both at the same time and just care for the dumb low latency, then you'll do one IO from each of them in turn. Latency will be good, but throughput will be aweful. And this means that in 20s they are both started, while with the slice idling and priority disk access that CFQ does, you'd hopefully have both up and running in 2s.
Quoted 6 lines
So latency is good, definitely, but sometimes you have to worry about the bigger picture too. Latency is more than single IOs, it's often for complete operation which may involve lots of IOs. Single IO latency is a benchmark thing, it's not a real life issue. And that's where it becomes complex and not so black and white. Mike's test is a really good example of that.
Quoted 3 lines
To the extent of you arguing that Mike's test is artificial (i'm not sure you are arguing that) - Mike certainly did not do an artificial test - he tested 'konsole' cache-cold startup latency, such as:
Quoted 6 lines
[snip]
I was saying the exact opposite, that Mike's test is a good example of a valid test. It's not measuring single IO latencies, it's doing a sequence of valid events and looking at the latency for those. It's benchmarking the bigger picture, not a microbenchmark.
Good, so we are in violent agreement :-)
Ingo
Quoted 27 lines · Ingo Molnar
Quoted 24 lines · Jens Axboe
Quoted 22 lines · Ingo Molnar
Quoted 9 lines · Jens Axboe
It's not _that_ easy, it depends a lot on the access patterns. A good example of that is actually the idling that we already do. Say you have two applications, each starting up. If you start them both at the same time and just care for the dumb low latency, then you'll do one IO from each of them in turn. Latency will be good, but throughput will be aweful. And this means that in 20s they are both started, while with the slice idling and priority disk access that CFQ does, you'd hopefully have both up and running in 2s.
Quoted 6 lines
So latency is good, definitely, but sometimes you have to worry about the bigger picture too. Latency is more than single IOs, it's often for complete operation which may involve lots of IOs. Single IO latency is a benchmark thing, it's not a real life issue. And that's where it becomes complex and not so black and white. Mike's test is a really good example of that.
To the extent of you arguing that Mike's test is artificial (i'm not sure you are arguing that) - Mike certainly did not do an artificial test - he tested 'konsole' cache-cold startup latency, such as:
Quoted 8 lines
Quoted 6 lines
[snip]
I was saying the exact opposite, that Mike's test is a good example of a valid test. It's not measuring single IO latencies, it's doing a sequence of valid events and looking at the latency for those. It's benchmarking the bigger picture, not a microbenchmark.
Good, so we are in violent agreement :-)
Yes, perhaps that last sentence didn't provide enough evidence of which category I put Mike's test into :-)
So to kick things off, I added an 'interactive' knob to CFQ and defaulted it to on, along with re-enabling slice idling for hardware that does tagged command queuing. This is almost completely identical to what Vivek Goyal originally posted, it's just combined into one and uses the term 'interactive' instead of 'fairness'. I think the former is a better umbrella under which to add further tweaks that may sacrifice throughput slightly, in the quest for better latency.
It's queued up in the for-linus branch.
Signature · 1 line
Jens Axboe
Quoted 48 lines · Jens Axboe
Quoted 27 lines · Ingo Molnar
Quoted 24 lines · Jens Axboe
Quoted 22 lines · Ingo Molnar
Quoted 9 lines · Jens Axboe
It's not _that_ easy, it depends a lot on the access patterns. A good example of that is actually the idling that we already do. Say you have two applications, each starting up. If you start them both at the same time and just care for the dumb low latency, then you'll do one IO from each of them in turn. Latency will be good, but throughput will be aweful. And this means that in 20s they are both started, while with the slice idling and priority disk access that CFQ does, you'd hopefully have both up and running in 2s.
Quoted 6 lines
So latency is good, definitely, but sometimes you have to worry about the bigger picture too. Latency is more than single IOs, it's often for complete operation which may involve lots of IOs. Single IO latency is a benchmark thing, it's not a real life issue. And that's where it becomes complex and not so black and white. Mike's test is a really good example of that.
To the extent of you arguing that Mike's test is artificial (i'm not sure you are arguing that) - Mike certainly did not do an artificial test - he tested 'konsole' cache-cold startup latency, such as:
Quoted 8 lines
Quoted 6 lines
[snip]
I was saying the exact opposite, that Mike's test is a good example of a valid test. It's not measuring single IO latencies, it's doing a sequence of valid events and looking at the latency for those. It's benchmarking the bigger picture, not a microbenchmark.
Good, so we are in violent agreement :-)
Yes, perhaps that last sentence didn't provide enough evidence of which category I put Mike's test into :-)
So to kick things off, I added an 'interactive' knob to CFQ and defaulted it to on, along with re-enabling slice idling for hardware that does tagged command queuing. This is almost completely identical to what Vivek Goyal originally posted, it's just combined into one and uses the term 'interactive' instead of 'fairness'. I think the former is a better umbrella under which to add further tweaks that may sacrifice throughput slightly, in the quest for better latency.
Quoted 1 line
It's queued up in the for-linus branch.
i'd say 'latency' describes it even better. 'interactivity' as a term is a bit overladen.
Ingo
Quoted 51 lines · Ingo Molnar
Quoted 48 lines · Jens Axboe
Quoted 27 lines · Ingo Molnar
Quoted 24 lines · Jens Axboe
Quoted 22 lines · Ingo Molnar
Quoted 9 lines · Jens Axboe
It's not _that_ easy, it depends a lot on the access patterns. A good example of that is actually the idling that we already do. Say you have two applications, each starting up. If you start them both at the same time and just care for the dumb low latency, then you'll do one IO from each of them in turn. Latency will be good, but throughput will be aweful. And this means that in 20s they are both started, while with the slice idling and priority disk access that CFQ does, you'd hopefully have both up and running in 2s.
Quoted 6 lines
So latency is good, definitely, but sometimes you have to worry about the bigger picture too. Latency is more than single IOs, it's often for complete operation which may involve lots of IOs. Single IO latency is a benchmark thing, it's not a real life issue. And that's where it becomes complex and not so black and white. Mike's test is a really good example of that.
To the extent of you arguing that Mike's test is artificial (i'm not sure you are arguing that) - Mike certainly did not do an artificial test - he tested 'konsole' cache-cold startup latency, such as:
Quoted 8 lines
Quoted 6 lines
[snip]
I was saying the exact opposite, that Mike's test is a good example of a valid test. It's not measuring single IO latencies, it's doing a sequence of valid events and looking at the latency for those. It's benchmarking the bigger picture, not a microbenchmark.
Good, so we are in violent agreement :-)
Yes, perhaps that last sentence didn't provide enough evidence of which category I put Mike's test into :-)
So to kick things off, I added an 'interactive' knob to CFQ and defaulted it to on, along with re-enabling slice idling for hardware that does tagged command queuing. This is almost completely identical to what Vivek Goyal originally posted, it's just combined into one and uses the term 'interactive' instead of 'fairness'. I think the former is a better umbrella under which to add further tweaks that may sacrifice throughput slightly, in the quest for better latency.
Quoted 4 lines
Quoted 1 line
It's queued up in the for-linus branch.
i'd say 'latency' describes it even better. 'interactivity' as a term is a bit overladen.
I'm not too crazy about it either. How about just using 'desktop' since this is obviously what we are really targetting? 'latency' isn't fully descriptive either, since it may not necessarily provide the best single IO latency (noop would).
Signature · 1 line
Jens Axboe
Quoted 4 lines · Jens Axboe
I'm not too crazy about it either. How about just using 'desktop' since this is obviously what we are really targetting? 'latency' isn't fully descriptive either, since it may not necessarily provide the best single IO latency (noop would).
Grin. "Perfect is the enemy of good" :)
Avg
16.24 175.82 154.38 228.97 147.16 144.5 noop
43.23 57.39 96.13 148.25 180.09 105.0 deadlineQuoted 11 lines · Mike Galbraith
Quoted 4 lines · Jens Axboe
I'm not too crazy about it either. How about just using 'desktop' since this is obviously what we are really targetting? 'latency' isn't fully descriptive either, since it may not necessarily provide the best single IO latency (noop would).
Grin. "Perfect is the enemy of good" :)
Avg 16.24 175.82 154.38 228.97 147.16 144.5 noop 43.23 57.39 96.13 148.25 180.09 105.0 deadline
Yep, that's where it falls down. Noop basically fails here because it treats all IO as equal, which obviously isn't true for most people. But even for pure read workloads (is the above the mixed read/write, or just read?), latency would be excellent with noop but the desktop experience would not.
Signature · 1 line
Jens Axboe
Quoted 13 lines · Jens Axboe
Quoted 11 lines · Mike Galbraith
Quoted 4 lines · Jens Axboe
I'm not too crazy about it either. How about just using 'desktop' since this is obviously what we are really targetting? 'latency' isn't fully descriptive either, since it may not necessarily provide the best single IO latency (noop would).
Grin. "Perfect is the enemy of good" :)
Avg 16.24 175.82 154.38 228.97 147.16 144.5 noop 43.23 57.39 96.13 148.25 180.09 105.0 deadline
Quoted 5 lines
Yep, that's where it falls down. Noop basically fails here because it treats all IO as equal, which obviously isn't true for most people. But even for pure read workloads (is the above the mixed read/write, or just read?), latency would be excellent with noop but the desktop experience would not.
Yeah, it's the dd vs konsole -e exit.
-Mike
A message that is not in the archive (sent off-list or not yet received): 20091002182608.GO31616-tSWWG44O7X1aa/[email protected]
Quoted 13 lines · Jens Axboe
Quoted 11 lines · Mike Galbraith
Quoted 4 lines · Jens Axboe
I'm not too crazy about it either. How about just using 'desktop' since this is obviously what we are really targetting? 'latency' isn't fully descriptive either, since it may not necessarily provide the best single IO latency (noop would).
Grin. "Perfect is the enemy of good" :)
Avg 16.24 175.82 154.38 228.97 147.16 144.5 noop 43.23 57.39 96.13 148.25 180.09 105.0 deadline
Quoted 5 lines
Yep, that's where it falls down. Noop basically fails here because it treats all IO as equal, which obviously isn't true for most people. But even for pure read workloads (is the above the mixed read/write, or just read?), latency would be excellent with noop but the desktop experience would not.
Yeah, it's the dd vs konsole -e exit.
-Mike
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 11 lines · Mike Galbraith
Quoted 4 lines · Jens Axboe
I'm not too crazy about it either. How about just using 'desktop' since this is obviously what we are really targetting? 'latency' isn't fully descriptive either, since it may not necessarily provide the best single IO latency (noop would).
Grin. "Perfect is the enemy of good" :)
Avg 16.24 175.82 154.38 228.97 147.16 144.5 noop 43.23 57.39 96.13 148.25 180.09 105.0 deadline
Yep, that's where it falls down. Noop basically fails here because it treats all IO as equal, which obviously isn't true for most people. But even for pure read workloads (is the above the mixed read/write, or just read?), latency would be excellent with noop but the desktop experience would not.
Signature · 1 line
Jens Axboe
A message that is not in the archive (sent off-list or not yet received): 20091002180437.GL31616-tSWWG44O7X1aa/[email protected]
Quoted 4 lines · Jens Axboe
I'm not too crazy about it either. How about just using 'desktop' since this is obviously what we are really targetting? 'latency' isn't fully descriptive either, since it may not necessarily provide the best single IO latency (noop would).
Grin. "Perfect is the enemy of good" :)
Avg
16.24 175.82 154.38 228.97 147.16 144.5 noop
43.23 57.39 96.13 148.25 180.09 105.0 deadlineQuoted 7 lines · Jens Axboe
Quoted 2 lines
i'd say 'latency' describes it even better. 'interactivity' as a term is a bit overladen.
I'm not too crazy about it either. How about just using 'desktop' since this is obviously what we are really targetting? 'latency' isn't fully descriptive either, since it may not necessarily provide the best single IO latency (noop would).
As Linus has already pointed out, it's not necessarily "desktop" versus "server". There will be certain high frequency transaction database workloads (for example) that will very much care about latency. I think "low_latency" may be the best term to use.
- Ted
Quoted 7 lines · Jens Axboe
Quoted 2 lines
i'd say 'latency' describes it even better. 'interactivity' as a term is a bit overladen.
I'm not too crazy about it either. How about just using 'desktop' since this is obviously what we are really targetting? 'latency' isn't fully descriptive either, since it may not necessarily provide the best single IO latency (noop would).
As Linus has already pointed out, it's not necessarily "desktop" versus "server". There will be certain high frequency transaction database workloads (for example) that will very much care about latency. I think "low_latency" may be the best term to use.
- Ted
Quoted 9 lines · Theodore Tso
Quoted 7 lines · Jens Axboe
Quoted 2 lines
i'd say 'latency' describes it even better. 'interactivity' as a term is a bit overladen.
I'm not too crazy about it either. How about just using 'desktop' since this is obviously what we are really targetting? 'latency' isn't fully descriptive either, since it may not necessarily provide the best single IO latency (noop would).
Quoted 4 lines
As Linus has already pointed out, it's not necessarily "desktop" versus "server". There will be certain high frequency transaction database workloads (for example) that will very much care about latency. I think "
low_latency" may be the best term to use.
Not necessarily, but typically it will be. As already noted, I don't think latency itself is a very descriptive term for this.
Signature · 1 line
Jens Axboe
Quoted 10 lines · Jens Axboe
Quoted 9 lines · Theodore Tso
Quoted 7 lines · Jens Axboe
Quoted 2 lines
i'd say 'latency' describes it even better. 'interactivity' as a term is a bit overladen.
I'm not too crazy about it either. How about just using 'desktop' since this is obviously what we are really targetting? 'latency' isn't fully descriptive either, since it may not necessarily provide the best single IO latency (noop would).
Quoted 7 lines
Quoted 4 lines
As Linus has already pointed out, it's not necessarily "desktop" versus "server". There will be certain high frequency transaction database workloads (for example) that will very much care about latency. I think "
low_latency" may be the best term to use.Not necessarily, but typically it will be. As already noted, I don't think latency itself is a very descriptive term for this.
Why not? Nobody will think of 'latency' as something that requires noop, but as something that in practice achieves low latencies, for stuff that people use.
Ingo
Quoted 18 lines · Ingo Molnar
Quoted 15 lines · Jens Axboe
Quoted 13 lines · Theodore Tso
Quoted 7 lines · Jens Axboe
Quoted 2 lines
i'd say 'latency' describes it even better. 'interactivity' as a term is a bit overladen.
I'm not too crazy about it either. How about just using 'desktop' since this is obviously what we are really targetting? 'latency' isn't fully descriptive either, since it may not necessarily provide the best single IO latency (noop would).
As Linus has already pointed out, it's not necessarily "desktop" versus "server". There will be certain high frequency transaction database workloads (for example) that will very much care about latency. I think "
low_latency" may be the best term to use.
Quoted 6 lines
Quoted 2 lines
Not necessarily, but typically it will be. As already noted, I don't think latency itself is a very descriptive term for this.
Why not? Nobody will think of 'latency' as something that requires noop, but as something that in practice achieves low latencies, for stuff that people use.
Alright, I'll acknowledge that if that's the general consensus. I may be somewhat biased myself.
Signature · 1 line
Jens Axboe
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 18 lines · Ingo Molnar
Quoted 15 lines · Jens Axboe
Quoted 13 lines · Theodore Tso
Quoted 7 lines · Jens Axboe
Quoted 2 lines
i'd say 'latency' describes it even better. 'interactivity' as a term is a bit overladen.
I'm not too crazy about it either. How about just using 'desktop' since this is obviously what we are really targetting? 'latency' isn't fully descriptive either, since it may not necessarily provide the best single IO latency (noop would).
As Linus has already pointed out, it's not necessarily "desktop" versus "server". There will be certain high frequency transaction database workloads (for example) that will very much care about latency. I think "
low_latency" may be the best term to use.
Quoted 6 lines
Quoted 2 lines
Not necessarily, but typically it will be. As already noted, I don't think latency itself is a very descriptive term for this.
Why not? Nobody will think of 'latency' as something that requires noop, but as something that in practice achieves low latencies, for stuff that people use.
Alright, I'll acknowledge that if that's the general consensus. I may be somewhat biased myself.
Signature · 1 line
Jens Axboe
A message that is not in the archive (sent off-list or not yet received): 20091002184549.GS31616-tSWWG44O7X1aa/[email protected]
Quoted 10 lines · Jens Axboe
Quoted 9 lines · Theodore Tso
Quoted 7 lines · Jens Axboe
Quoted 2 lines
i'd say 'latency' describes it even better. 'interactivity' as a term is a bit overladen.
I'm not too crazy about it either. How about just using 'desktop' since this is obviously what we are really targetting? 'latency' isn't fully descriptive either, since it may not necessarily provide the best single IO latency (noop would).
Quoted 7 lines
Quoted 4 lines
As Linus has already pointed out, it's not necessarily "desktop" versus "server". There will be certain high frequency transaction database workloads (for example) that will very much care about latency. I think "
low_latency" may be the best term to use.Not necessarily, but typically it will be. As already noted, I don't think latency itself is a very descriptive term for this.
Why not? Nobody will think of 'latency' as something that requires noop, but as something that in practice achieves low latencies, for stuff that people use.
Ingo
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 9 lines · Theodore Tso
Quoted 7 lines · Jens Axboe
Quoted 2 lines
i'd say 'latency' describes it even better. 'interactivity' as a term is a bit overladen.
I'm not too crazy about it either. How about just using 'desktop' since this is obviously what we are really targetting? 'latency' isn't fully descriptive either, since it may not necessarily provide the best single IO latency (noop would).
Quoted 4 lines
As Linus has already pointed out, it's not necessarily "desktop" versus "server". There will be certain high frequency transaction database workloads (for example) that will very much care about latency. I think "
low_latency" may be the best term to use.
Not necessarily, but typically it will be. As already noted, I don't think latency itself is a very descriptive term for this.
Signature · 1 line
Jens Axboe
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 51 lines · Ingo Molnar
Quoted 48 lines · Jens Axboe
Quoted 27 lines · Ingo Molnar
Quoted 24 lines · Jens Axboe
Quoted 22 lines · Ingo Molnar
Quoted 9 lines · Jens Axboe
It's not _that_ easy, it depends a lot on the access patterns. A good example of that is actually the idling that we already do. Say you have two applications, each starting up. If you start them both at the same time and just care for the dumb low latency, then you'll do one IO from each of them in turn. Latency will be good, but throughput will be aweful. And this means that in 20s they are both started, while with the slice idling and priority disk access that CFQ does, you'd hopefully have both up and running in 2s.
Quoted 6 lines
So latency is good, definitely, but sometimes you have to worry about the bigger picture too. Latency is more than single IOs, it's often for complete operation which may involve lots of IOs. Single IO latency is a benchmark thing, it's not a real life issue. And that's where it becomes complex and not so black and white. Mike's test is a really good example of that.
To the extent of you arguing that Mike's test is artificial (i'm not sure you are arguing that) - Mike certainly did not do an artificial test - he tested 'konsole' cache-cold startup latency, such as:
Quoted 8 lines
Quoted 6 lines
[snip]
I was saying the exact opposite, that Mike's test is a good example of a valid test. It's not measuring single IO latencies, it's doing a sequence of valid events and looking at the latency for those. It's benchmarking the bigger picture, not a microbenchmark.
Good, so we are in violent agreement :-)
Yes, perhaps that last sentence didn't provide enough evidence of which category I put Mike's test into :-)
So to kick things off, I added an 'interactive' knob to CFQ and defaulted it to on, along with re-enabling slice idling for hardware that does tagged command queuing. This is almost completely identical to what Vivek Goyal originally posted, it's just combined into one and uses the term 'interactive' instead of 'fairness'. I think the former is a better umbrella under which to add further tweaks that may sacrifice throughput slightly, in the quest for better latency.
Quoted 4 lines
Quoted 1 line
It's queued up in the for-linus branch.
i'd say 'latency' describes it even better. 'interactivity' as a term is a bit overladen.
I'm not too crazy about it either. How about just using 'desktop' since this is obviously what we are really targetting? 'latency' isn't fully descriptive either, since it may not necessarily provide the best single IO latency (noop would).
Signature · 1 line
Jens Axboe
A message that is not in the archive (sent off-list or not yet received): 20091002173732.GK31616-tSWWG44O7X1aa/[email protected]
Quoted 48 lines · Jens Axboe
Quoted 27 lines · Ingo Molnar
Quoted 24 lines · Jens Axboe
Quoted 22 lines · Ingo Molnar
Quoted 9 lines · Jens Axboe
It's not _that_ easy, it depends a lot on the access patterns. A good example of that is actually the idling that we already do. Say you have two applications, each starting up. If you start them both at the same time and just care for the dumb low latency, then you'll do one IO from each of them in turn. Latency will be good, but throughput will be aweful. And this means that in 20s they are both started, while with the slice idling and priority disk access that CFQ does, you'd hopefully have both up and running in 2s.
Quoted 6 lines
So latency is good, definitely, but sometimes you have to worry about the bigger picture too. Latency is more than single IOs, it's often for complete operation which may involve lots of IOs. Single IO latency is a benchmark thing, it's not a real life issue. And that's where it becomes complex and not so black and white. Mike's test is a really good example of that.
To the extent of you arguing that Mike's test is artificial (i'm not sure you are arguing that) - Mike certainly did not do an artificial test - he tested 'konsole' cache-cold startup latency, such as:
Quoted 8 lines
Quoted 6 lines
[snip]
I was saying the exact opposite, that Mike's test is a good example of a valid test. It's not measuring single IO latencies, it's doing a sequence of valid events and looking at the latency for those. It's benchmarking the bigger picture, not a microbenchmark.
Good, so we are in violent agreement :-)
Yes, perhaps that last sentence didn't provide enough evidence of which category I put Mike's test into :-)
So to kick things off, I added an 'interactive' knob to CFQ and defaulted it to on, along with re-enabling slice idling for hardware that does tagged command queuing. This is almost completely identical to what Vivek Goyal originally posted, it's just combined into one and uses the term 'interactive' instead of 'fairness'. I think the former is a better umbrella under which to add further tweaks that may sacrifice throughput slightly, in the quest for better latency.
Quoted 1 line
It's queued up in the for-linus branch.
i'd say 'latency' describes it even better. 'interactivity' as a term is a bit overladen.
Ingo
Quoted 48 lines · Jens Axboe
Quoted 27 lines · Ingo Molnar
Quoted 24 lines · Jens Axboe
Quoted 22 lines · Ingo Molnar
Quoted 9 lines · Jens Axboe
It's not _that_ easy, it depends a lot on the access patterns. A good example of that is actually the idling that we already do. Say you have two applications, each starting up. If you start them both at the same time and just care for the dumb low latency, then you'll do one IO from each of them in turn. Latency will be good, but throughput will be aweful. And this means that in 20s they are both started, while with the slice idling and priority disk access that CFQ does, you'd hopefully have both up and running in 2s.
Quoted 6 lines
So latency is good, definitely, but sometimes you have to worry about the bigger picture too. Latency is more than single IOs, it's often for complete operation which may involve lots of IOs. Single IO latency is a benchmark thing, it's not a real life issue. And that's where it becomes complex and not so black and white. Mike's test is a really good example of that.
To the extent of you arguing that Mike's test is artificial (i'm not sure you are arguing that) - Mike certainly did not do an artificial test - he tested 'konsole' cache-cold startup latency, such as:
Quoted 8 lines
Quoted 6 lines
[snip]
I was saying the exact opposite, that Mike's test is a good example of a valid test. It's not measuring single IO latencies, it's doing a sequence of valid events and looking at the latency for those. It's benchmarking the bigger picture, not a microbenchmark.
Good, so we are in violent agreement :-)
Yes, perhaps that last sentence didn't provide enough evidence of which category I put Mike's test into :-)
So to kick things off, I added an 'interactive' knob to CFQ and defaulted it to on, along with re-enabling slice idling for hardware that does tagged command queuing. This is almost completely identical to what Vivek Goyal originally posted, it's just combined into one and uses the term 'interactive' instead of 'fairness'. I think the former is a better umbrella under which to add further tweaks that may sacrifice throughput slightly, in the quest for better latency.
Quoted 1 line
It's queued up in the for-linus branch.
FWIW, I did a matrix of Vivek's patch combined with my hack. Seems we do lose a bit of dd throughput over stock with either or both.
dd pre 65.1 65.4 67.5 64.8 65.1 65.5 fairness=1 overload_delay=1 perf stat 1.70 1.94 1.32 1.89 1.87 1.7 dd post 69.4 62.3 69.7 70.3 69.6 68.2
dd pre 67.0 67.8 64.7 64.7 64.9 65.8 fairness=1 overload_delay=0 perf stat 4.89 3.13 2.98 2.71 2.17 3.1 dd post 67.2 63.3 62.6 62.8 63.1 63.8
dd pre 65.0 66.0 66.9 64.6 67.0 65.9 fairness=0 overload_delay=1 perf stat 4.66 3.81 4.23 2.98 4.23 3.9 dd post 62.0 60.8 62.4 61.4 62.2 61.7
dd pre 65.3 65.6 64.9 69.5 65.8 66.2 fairness=0 overload_delay=0 perf stat 14.79 9.11 14.16 8.44 13.67 12.0 dd post 64.1 66.5 64.0 66.5 64.4 65.1
Quoted 48 lines · Jens Axboe
Quoted 27 lines · Ingo Molnar
Quoted 24 lines · Jens Axboe
Quoted 22 lines · Ingo Molnar
Quoted 9 lines · Jens Axboe
It's not _that_ easy, it depends a lot on the access patterns. A good example of that is actually the idling that we already do. Say you have two applications, each starting up. If you start them both at the same time and just care for the dumb low latency, then you'll do one IO from each of them in turn. Latency will be good, but throughput will be aweful. And this means that in 20s they are both started, while with the slice idling and priority disk access that CFQ does, you'd hopefully have both up and running in 2s.
Quoted 6 lines
So latency is good, definitely, but sometimes you have to worry about the bigger picture too. Latency is more than single IOs, it's often for complete operation which may involve lots of IOs. Single IO latency is a benchmark thing, it's not a real life issue. And that's where it becomes complex and not so black and white. Mike's test is a really good example of that.
To the extent of you arguing that Mike's test is artificial (i'm not sure you are arguing that) - Mike certainly did not do an artificial test - he tested 'konsole' cache-cold startup latency, such as:
Quoted 8 lines
Quoted 6 lines
[snip]
I was saying the exact opposite, that Mike's test is a good example of a valid test. It's not measuring single IO latencies, it's doing a sequence of valid events and looking at the latency for those. It's benchmarking the bigger picture, not a microbenchmark.
Good, so we are in violent agreement :-)
Yes, perhaps that last sentence didn't provide enough evidence of which category I put Mike's test into :-)
So to kick things off, I added an 'interactive' knob to CFQ and defaulted it to on, along with re-enabling slice idling for hardware that does tagged command queuing. This is almost completely identical to what Vivek Goyal originally posted, it's just combined into one and uses the term 'interactive' instead of 'fairness'. I think the former is a better umbrella under which to add further tweaks that may sacrifice throughput slightly, in the quest for better latency.
Quoted 1 line
It's queued up in the for-linus branch.
FWIW, I did a matrix of Vivek's patch combined with my hack. Seems we do lose a bit of dd throughput over stock with either or both.
dd pre 65.1 65.4 67.5 64.8 65.1 65.5 fairness=1 overload_delay=1 perf stat 1.70 1.94 1.32 1.89 1.87 1.7 dd post 69.4 62.3 69.7 70.3 69.6 68.2
dd pre 67.0 67.8 64.7 64.7 64.9 65.8 fairness=1 overload_delay=0 perf stat 4.89 3.13 2.98 2.71 2.17 3.1 dd post 67.2 63.3 62.6 62.8 63.1 63.8
dd pre 65.0 66.0 66.9 64.6 67.0 65.9 fairness=0 overload_delay=1 perf stat 4.66 3.81 4.23 2.98 4.23 3.9 dd post 62.0 60.8 62.4 61.4 62.2 61.7
dd pre 65.3 65.6 64.9 69.5 65.8 66.2 fairness=0 overload_delay=0 perf stat 14.79 9.11 14.16 8.44 13.67 12.0 dd post 64.1 66.5 64.0 66.5 64.4 65.1
Quoted 62 lines · Mike Galbraith
Quoted 48 lines · Jens Axboe
Quoted 27 lines · Ingo Molnar
Quoted 24 lines · Jens Axboe
Quoted 22 lines · Ingo Molnar
Quoted 9 lines · Jens Axboe
It's not _that_ easy, it depends a lot on the access patterns. A good example of that is actually the idling that we already do. Say you have two applications, each starting up. If you start them both at the same time and just care for the dumb low latency, then you'll do one IO from each of them in turn. Latency will be good, but throughput will be aweful. And this means that in 20s they are both started, while with the slice idling and priority disk access that CFQ does, you'd hopefully have both up and running in 2s.
Quoted 6 lines
So latency is good, definitely, but sometimes you have to worry about the bigger picture too. Latency is more than single IOs, it's often for complete operation which may involve lots of IOs. Single IO latency is a benchmark thing, it's not a real life issue. And that's where it becomes complex and not so black and white. Mike's test is a really good example of that.
To the extent of you arguing that Mike's test is artificial (i'm not sure you are arguing that) - Mike certainly did not do an artificial test - he tested 'konsole' cache-cold startup latency, such as:
Quoted 8 lines
Quoted 6 lines
[snip]
I was saying the exact opposite, that Mike's test is a good example of a valid test. It's not measuring single IO latencies, it's doing a sequence of valid events and looking at the latency for those. It's benchmarking the bigger picture, not a microbenchmark.
Good, so we are in violent agreement :-)
Yes, perhaps that last sentence didn't provide enough evidence of which category I put Mike's test into :-)
So to kick things off, I added an 'interactive' knob to CFQ and defaulted it to on, along with re-enabling slice idling for hardware that does tagged command queuing. This is almost completely identical to what Vivek Goyal originally posted, it's just combined into one and uses the term 'interactive' instead of 'fairness'. I think the former is a better umbrella under which to add further tweaks that may sacrifice throughput slightly, in the quest for better latency.
Quoted 1 line
It's queued up in the for-linus branch.
FWIW, I did a matrix of Vivek's patch combined with my hack. Seems we do lose a bit of dd throughput over stock with either or both.
dd pre 65.1 65.4 67.5 64.8 65.1 65.5 fairness=1 overload_delay=1 perf stat 1.70 1.94 1.32 1.89 1.87 1.7 dd post 69.4 62.3 69.7 70.3 69.6 68.2dd pre 67.0 67.8 64.7 64.7 64.9 65.8 fairness=1 overload_delay=0 perf stat 4.89 3.13 2.98 2.71 2.17 3.1 dd post 67.2 63.3 62.6 62.8 63.1 63.8
Quoted 7 lines
dd pre 65.0 66.0 66.9 64.6 67.0 65.9 fairness=0 overload_delay=1 perf stat 4.66 3.81 4.23 2.98 4.23 3.9 dd post 62.0 60.8 62.4 61.4 62.2 61.7dd pre 65.3 65.6 64.9 69.5 65.8 66.2 fairness=0 overload_delay=0 perf stat 14.79 9.11 14.16 8.44 13.67 12.0 dd post 64.1 66.5 64.0 66.5 64.4 65.1
I'm not too worried about the "single IO producer" scenarios, and it looks like (from a quick look) that most of your numbers are within some expected noise levels. It's the more complex mixes that are likely to cause a bit of a stink, but lets worry about that later. One quick thing would be to read eg 2 or more files sequentially from disk and see how that performs.
If you could do a cleaned up version of your overload patch based on this:
http://git.kernel.dk/?p=linux-2.6-block.git;a=commit;h=1d2235152dc745c6d94bedb550fea84cffdbf768
then lets take it from there.
Signature · 1 line
Jens Axboe
Quoted 6 lines · Jens Axboe
I'm not too worried about the "single IO producer" scenarios, and it looks like (from a quick look) that most of your numbers are within some expected noise levels. It's the more complex mixes that are likely to cause a bit of a stink, but lets worry about that later. One quick thing would be to read eg 2 or more files sequentially from disk and see how that performs.
Hm. git(s) should be good for a nice repeatable load. Suggestions?
Quoted 6 lines
If you could do a cleaned up version of your overload patch based on this:
http://git.kernel.dk/?p=linux-2.6-block.git;a=commit;h=1d2235152dc745c6d94bedb550fea84cffdbf768
then lets take it from there.
I'll try to find a good repeatable git beater first. At this point, I only know it helps with one load.
-Mike
Quoted 14 lines · Mike Galbraith
Quoted 6 lines · Jens Axboe
I'm not too worried about the "single IO producer" scenarios, and it looks like (from a quick look) that most of your numbers are within some expected noise levels. It's the more complex mixes that are likely to cause a bit of a stink, but lets worry about that later. One quick thing would be to read eg 2 or more files sequentially from disk and see how that performs.
Hm. git(s) should be good for a nice repeatable load. Suggestions?
Quoted 3 lines
If you could do a cleaned up version of your overload patch based on this:
Quoted 6 lines
Quoted 3 lines
http://git.kernel.dk/?p=linux-2.6-block.git;a=commit;h=1d2235152dc745c6d94bedb550fea84cffdbf768
then lets take it from there.
I'll try to find a good repeatable git beater first. At this point, I only know it helps with one load.
Seems to help mixed concurrent read/write a bit too.
perf stat testo.sh Avg
108.12 106.33 106.34 97.00 106.52 104.8 1.000 fairness=0 overload_delay=0 93.98 102.44 94.47 97.70 98.90 97.4 .929 fairness=0 overload_delay=1 90.87 95.40 95.79 93.09 94.25 93.8 .895 fairness=1 overload_delay=0 89.93 90.57 89.13 93.43 93.72 91.3 .871 fairness=1 overload_delay=1
#!/bin/sh
LOGFILE=testo.log
rm -f $LOGFILE
echo 3 > /proc/sys/vm/drop_caches
sh -c "(cd linux-2.6.23; perf stat -- git checkout -f; git archive --format=tar HEAD > ../linux-2.6.23.tar)" 2>&1|tee -a $LOGFILE & sh -c "(cd linux-2.6.24; perf stat -- git archive --format=tar HEAD > ../linux-2.6.24.tar; git checkout -f)" 2>&1|tee -a $LOGFILE & sh -c "(cd linux-2.6.25; perf stat -- git checkout -f; git archive --format=tar HEAD > ../linux-2.6.25.tar)" 2>&1|tee -a $LOGFILE & sh -c "(cd linux-2.6.26; perf stat -- git archive --format=tar HEAD > ../linux-2.6.26.tar; git checkout -f)" 2>&1|tee -a $LOGFILE &
wait
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 14 lines · Mike Galbraith
Quoted 6 lines · Jens Axboe
I'm not too worried about the "single IO producer" scenarios, and it looks like (from a quick look) that most of your numbers are within some expected noise levels. It's the more complex mixes that are likely to cause a bit of a stink, but lets worry about that later. One quick thing would be to read eg 2 or more files sequentially from disk and see how that performs.
Hm. git(s) should be good for a nice repeatable load. Suggestions?
Quoted 3 lines
If you could do a cleaned up version of your overload patch based on this:
Quoted 6 lines
Quoted 3 lines
http://git.kernel.dk/?p=linux-2.6-block.git;a=commit;h=1d2235152dc745c6d94bedb550fea84cffdbf768
then lets take it from there.
I'll try to find a good repeatable git beater first. At this point, I only know it helps with one load.
Seems to help mixed concurrent read/write a bit too.
perf stat testo.sh Avg
108.12 106.33 106.34 97.00 106.52 104.8 1.000 fairness=0 overload_delay=0 93.98 102.44 94.47 97.70 98.90 97.4 .929 fairness=0 overload_delay=1 90.87 95.40 95.79 93.09 94.25 93.8 .895 fairness=1 overload_delay=0 89.93 90.57 89.13 93.43 93.72 91.3 .871 fairness=1 overload_delay=1
#!/bin/sh
LOGFILE=testo.log
rm -f $LOGFILE
echo 3 > /proc/sys/vm/drop_caches
sh -c "(cd linux-2.6.23; perf stat -- git checkout -f; git archive --format=tar HEAD > ../linux-2.6.23.tar)" 2>&1|tee -a $LOGFILE & sh -c "(cd linux-2.6.24; perf stat -- git archive --format=tar HEAD > ../linux-2.6.24.tar; git checkout -f)" 2>&1|tee -a $LOGFILE & sh -c "(cd linux-2.6.25; perf stat -- git checkout -f; git archive --format=tar HEAD > ../linux-2.6.25.tar)" 2>&1|tee -a $LOGFILE & sh -c "(cd linux-2.6.26; perf stat -- git archive --format=tar HEAD > ../linux-2.6.26.tar; git checkout -f)" 2>&1|tee -a $LOGFILE &
wait
A message that is not in the archive (sent off-list or not yet received): 20091002181903.GN31616-tSWWG44O7X1aa/[email protected]
Quoted 6 lines · Jens Axboe
I'm not too worried about the "single IO producer" scenarios, and it looks like (from a quick look) that most of your numbers are within some expected noise levels. It's the more complex mixes that are likely to cause a bit of a stink, but lets worry about that later. One quick thing would be to read eg 2 or more files sequentially from disk and see how that performs.
Hm. git(s) should be good for a nice repeatable load. Suggestions?
Quoted 6 lines
If you could do a cleaned up version of your overload patch based on this:
http://git.kernel.dk/?p=linux-2.6-block.git;a=commit;h=1d2235152dc745c6d94bedb550fea84cffdbf768
then lets take it from there.
I'll try to find a good repeatable git beater first. At this point, I only know it helps with one load.
-Mike
Quoted 6 lines · Jens Axboe
If you could do a cleaned up version of your overload patch based on this:
http://git.kernel.dk/?p=linux-2.6-block.git;a=commit;h=1d2235152dc745c6d94bedb550fea84cffdbf768
then lets take it from there.
If take it from there ends up meaning apply, and see who squeaks, feel free to delete the "Not", and my somewhat defective sense of humor.
Block: Delay overloading of CFQ queues to improve read latency.
Introduce a delay maximum dispatch timestamp, and stamp it when:
1. we encounter a known seeky or possibly new sync IO queue.
2. the current queue may go idle and we're draining async IO.
3. we have sync IO in flight and are servicing an async queue. 4 we are not the sole user of disk.
Disallow exceeding quantum if any of these events have occurred recently.
Protect this behavioral change with a "desktop_dispatch" knob and default it to "on".. providing an easy means of regression verification prior to hate-mail dispatch :) to CC list.
... others who let somewhat hacky tweak slip by
LKML-Reference: <new-submission>
--- block/cfq-iosched.c | 45 +++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 41 insertions(+), 4 deletions(-)
block/cfq-iosched.c +41 -4
@@ -174,6 +174,9 @@ struct cfq_data {unsigned int cfq_slice_async_rq;unsigned int cfq_slice_idle;unsigned int cfq_desktop; +unsigned int cfq_desktop_dispatch; + +unsigned long desktop_dispatch_ts;struct list_head cic_list;
@@ -1283,6 +1286,7 @@ static int cfq_dispatch_requests(structstruct cfq_data *cfqd = q->elevator->elevator_data;struct cfq_queue *cfqq;unsigned int max_dispatch; +unsigned long delay;if (!cfqd->busy_queues)return 0 ;
@@ -1297,19 +1301,26 @@ static int cfq_dispatch_requests(struct/* * Drain async requests before we start sync IO */ -if (cfq_cfqq_idle_window (cfqq) && cfqd->rq_in_driver[BLK_RW_ASYNC ]) +if (cfq_cfqq_idle_window (cfqq) && cfqd->rq_in_driver[BLK_RW_ASYNC ]) { + cfqd->desktop_dispatch_ts = jiffies;return 0 ; + }/* * If this is an async queue and we have sync IO in flight, let it wait */ -if (cfqd->sync_flight && !cfq_cfqq_sync (cfqq)) +if (cfqd->sync_flight && !cfq_cfqq_sync (cfqq)) { + cfqd->desktop_dispatch_ts = jiffies;return 0 ; + } max_dispatch = cfqd->cfq_quantum;if (cfq_class_idle (cfqq)) max_dispatch =1 ; +if (cfqd->busy_queues >1 ) + cfqd->desktop_dispatch_ts = jiffies; +/* * Does this cfqq already have too much IO in flight? */
@@ -1327,6 +1338,16 @@ static int cfq_dispatch_requests(structreturn 0 ;/* +* Don't start overloading until we've been alone for a bit. +*/ +if (cfqd->cfq_desktop_dispatch) { + delay = cfqd->desktop_dispatch_ts + cfq_slice_sync; + +if (time_before (jiffies, max_delay)) +return 0 ; + } + +/* * we are the only queue, allow up to 4 times of 'quantum' */ if (cfqq->dispatched >=4 * max_dispatch)
@@ -1942,7 +1963,7 @@ static void cfq_update_idle_window(struct cfq_data *cfqd, struct cfq_queue *cfqq, struct cfq_io_context *cic) { - int old_idle, enable_idle; + int old_idle, enable_idle, seeky = 0;/* * Don't idle for async or idle io prio class
@@ -1950,10 +1971,20 @@ cfq_update_idle_window(struct cfq_data *if (!cfq_cfqq_sync (cfqq) ||cfq_class_idle (cfqq))return ; +if (cfqd->hw_tag) { +if (CIC_SEEKY (cic)) + seeky =1 ; +/* +* If seeky or incalculable seekiness, delay overloading. +*/ +if (seeky || !sample_valid (cic->seek_samples)) + cfqd->desktop_dispatch_ts = jiffies; + } + enable_idle = old_idle =cfq_cfqq_idle_window (cfqq);if (!atomic_read (&cic->ioc->nr_tasks) || !cfqd->cfq_slice_idle || - (!cfqd->cfq_desktop && cfqd->hw_tag &&CIC_SEEKY (cic))) + (!cfqd->cfq_desktop && seeky)) enable_idle =0 ;else if (sample_valid (cic->ttime_samples)) {if (cic->ttime_mean > cfqd->cfq_slice_idle)
@@ -2483,6 +2514,9 @@ static void *cfq_init_queue(struct reque cfqd->cfq_slice_async_rq = cfq_slice_async_rq; cfqd->cfq_slice_idle = cfq_slice_idle; cfqd->cfq_desktop =1 ; + cfqd->cfq_desktop_dispatch =1 ; + + cfqd->desktop_dispatch_ts =INITIAL_JIFFIES ; cfqd->hw_tag =1 ;return cfqd;
@@ -2553,6 +2587,7 @@ SHOW_FUNCTION(cfq_slice_sync_show, cfqd-SHOW_FUNCTION (cfq_slice_async_show, cfqd->cfq_slice[0 ],1 );SHOW_FUNCTION (cfq_slice_async_rq_show, cfqd->cfq_slice_async_rq,0 );SHOW_FUNCTION (cfq_desktop_show, cfqd->cfq_desktop,0 ); +SHOW_FUNCTION (cfq_desktop_dispatch_show, cfqd->cfq_desktop_dispatch,0 );#undef SHOW_FUNCTION#define STORE_FUNCTION (__FUNC, __PTR,MIN ,MAX , __CONV) \
@@ -2585,6 +2620,7 @@ STORE_FUNCTION(cfq_slice_async_store, &cSTORE_FUNCTION (cfq_slice_async_rq_store, &cfqd->cfq_slice_async_rq,1 ,UINT_MAX ,0 );STORE_FUNCTION (cfq_desktop_store, &cfqd->cfq_desktop,0 ,1 ,0 ); +STORE_FUNCTION (cfq_desktop_dispatch_store, &cfqd->cfq_desktop_dispatch,0 ,1 ,0 );#undef STORE_FUNCTION#define CFQ_ATTR (name) \
@@ -2601,6 +2637,7 @@ static struct elv_fs_entry cfq_attrs[] =CFQ_ATTR (slice_async_rq),CFQ_ATTR (slice_idle),CFQ_ATTR (desktop), +CFQ_ATTR (desktop_dispatch), __ATTR_NULL };
Quoted 6 lines · Jens Axboe
If you could do a cleaned up version of your overload patch based on this:
http://git.kernel.dk/?p=linux-2.6-block.git;a=commit;h=1d2235152dc745c6d94bedb550fea84cffdbf768
then lets take it from there.
If take it from there ends up meaning apply, and see who squeaks, feel free to delete the "Not", and my somewhat defective sense of humor.
Block: Delay overloading of CFQ queues to improve read latency.
Introduce a delay maximum dispatch timestamp, and stamp it when:
1. we encounter a known seeky or possibly new sync IO queue.
2. the current queue may go idle and we're draining async IO.
3. we have sync IO in flight and are servicing an async queue. 4 we are not the sole user of disk.
Disallow exceeding quantum if any of these events have occurred recently.
Protect this behavioral change with a "desktop_dispatch" knob and default it to "on".. providing an easy means of regression verification prior to hate-mail dispatch :) to CC list.
... others who let somewhat hacky tweak slip by
LKML-Reference: <new-submission>
--- block/cfq-iosched.c | 45 +++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 41 insertions(+), 4 deletions(-)
block/cfq-iosched.c +41 -4
@@ -174,6 +174,9 @@ struct cfq_data {unsigned int cfq_slice_async_rq;unsigned int cfq_slice_idle;unsigned int cfq_desktop; +unsigned int cfq_desktop_dispatch; + +unsigned long desktop_dispatch_ts;struct list_head cic_list;
@@ -1283,6 +1286,7 @@ static int cfq_dispatch_requests(structstruct cfq_data *cfqd = q->elevator->elevator_data;struct cfq_queue *cfqq;unsigned int max_dispatch; +unsigned long delay;if (!cfqd->busy_queues)return 0 ;
@@ -1297,19 +1301,26 @@ static int cfq_dispatch_requests(struct/* * Drain async requests before we start sync IO */ -if (cfq_cfqq_idle_window (cfqq) && cfqd->rq_in_driver[BLK_RW_ASYNC ]) +if (cfq_cfqq_idle_window (cfqq) && cfqd->rq_in_driver[BLK_RW_ASYNC ]) { + cfqd->desktop_dispatch_ts = jiffies;return 0 ; + }/* * If this is an async queue and we have sync IO in flight, let it wait */ -if (cfqd->sync_flight && !cfq_cfqq_sync (cfqq)) +if (cfqd->sync_flight && !cfq_cfqq_sync (cfqq)) { + cfqd->desktop_dispatch_ts = jiffies;return 0 ; + } max_dispatch = cfqd->cfq_quantum;if (cfq_class_idle (cfqq)) max_dispatch =1 ; +if (cfqd->busy_queues >1 ) + cfqd->desktop_dispatch_ts = jiffies; +/* * Does this cfqq already have too much IO in flight? */
@@ -1327,6 +1338,16 @@ static int cfq_dispatch_requests(structreturn 0 ;/* +* Don't start overloading until we've been alone for a bit. +*/ +if (cfqd->cfq_desktop_dispatch) { + delay = cfqd->desktop_dispatch_ts + cfq_slice_sync; + +if (time_before (jiffies, max_delay)) +return 0 ; + } + +/* * we are the only queue, allow up to 4 times of 'quantum' */ if (cfqq->dispatched >=4 * max_dispatch)
@@ -1942,7 +1963,7 @@ static void cfq_update_idle_window(struct cfq_data *cfqd, struct cfq_queue *cfqq, struct cfq_io_context *cic) { - int old_idle, enable_idle; + int old_idle, enable_idle, seeky = 0;/* * Don't idle for async or idle io prio class
@@ -1950,10 +1971,20 @@ cfq_update_idle_window(struct cfq_data *if (!cfq_cfqq_sync (cfqq) ||cfq_class_idle (cfqq))return ; +if (cfqd->hw_tag) { +if (CIC_SEEKY (cic)) + seeky =1 ; +/* +* If seeky or incalculable seekiness, delay overloading. +*/ +if (seeky || !sample_valid (cic->seek_samples)) + cfqd->desktop_dispatch_ts = jiffies; + } + enable_idle = old_idle =cfq_cfqq_idle_window (cfqq);if (!atomic_read (&cic->ioc->nr_tasks) || !cfqd->cfq_slice_idle || - (!cfqd->cfq_desktop && cfqd->hw_tag &&CIC_SEEKY (cic))) + (!cfqd->cfq_desktop && seeky)) enable_idle =0 ;else if (sample_valid (cic->ttime_samples)) {if (cic->ttime_mean > cfqd->cfq_slice_idle)
@@ -2483,6 +2514,9 @@ static void *cfq_init_queue(struct reque cfqd->cfq_slice_async_rq = cfq_slice_async_rq; cfqd->cfq_slice_idle = cfq_slice_idle; cfqd->cfq_desktop =1 ; + cfqd->cfq_desktop_dispatch =1 ; + + cfqd->desktop_dispatch_ts =INITIAL_JIFFIES ; cfqd->hw_tag =1 ;return cfqd;
@@ -2553,6 +2587,7 @@ SHOW_FUNCTION(cfq_slice_sync_show, cfqd-SHOW_FUNCTION (cfq_slice_async_show, cfqd->cfq_slice[0 ],1 );SHOW_FUNCTION (cfq_slice_async_rq_show, cfqd->cfq_slice_async_rq,0 );SHOW_FUNCTION (cfq_desktop_show, cfqd->cfq_desktop,0 ); +SHOW_FUNCTION (cfq_desktop_dispatch_show, cfqd->cfq_desktop_dispatch,0 );#undef SHOW_FUNCTION#define STORE_FUNCTION (__FUNC, __PTR,MIN ,MAX , __CONV) \
@@ -2585,6 +2620,7 @@ STORE_FUNCTION(cfq_slice_async_store, &cSTORE_FUNCTION (cfq_slice_async_rq_store, &cfqd->cfq_slice_async_rq,1 ,UINT_MAX ,0 );STORE_FUNCTION (cfq_desktop_store, &cfqd->cfq_desktop,0 ,1 ,0 ); +STORE_FUNCTION (cfq_desktop_dispatch_store, &cfqd->cfq_desktop_dispatch,0 ,1 ,0 );#undef STORE_FUNCTION#define CFQ_ATTR (name) \
@@ -2601,6 +2637,7 @@ static struct elv_fs_entry cfq_attrs[] =CFQ_ATTR (slice_async_rq),CFQ_ATTR (slice_idle),CFQ_ATTR (desktop), +CFQ_ATTR (desktop_dispatch), __ATTR_NULL };
Quoted 8 lines · Mike Galbraith
Quoted 6 lines · Jens Axboe
If you could do a cleaned up version of your overload patch based on this:
http://git.kernel.dk/?p=linux-2.6-block.git;a=commit;h=1d2235152dc745c6d94bedb550fea84cffdbf768
then lets take it from there.
Note to self: build the darn thing after last minute changes.
Block: Delay overloading of CFQ queues to improve read latency.
Introduce a delay maximum dispatch timestamp, and stamp it when:
1. we encounter a known seeky or possibly new sync IO queue.
2. the current queue may go idle and we're draining async IO.
3. we have sync IO in flight and are servicing an async queue. 4 we are not the sole user of disk.
Disallow exceeding quantum if any of these events have occurred recently.
Protect this behavioral change with a "desktop_dispatch" knob and default it to "on".. providing an easy means of regression verification prior to hate-mail dispatch :) to CC list.
... others who let somewhat hacky tweak slip by
--- block/cfq-iosched.c | 45 +++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 41 insertions(+), 4 deletions(-)
block/cfq-iosched.c +41 -4
@@ -174,6 +174,9 @@ struct cfq_data {unsigned int cfq_slice_async_rq;unsigned int cfq_slice_idle;unsigned int cfq_desktop; +unsigned int cfq_desktop_dispatch; + +unsigned long desktop_dispatch_ts;struct list_head cic_list;
@@ -1283,6 +1286,7 @@ static int cfq_dispatch_requests(structstruct cfq_data *cfqd = q->elevator->elevator_data;struct cfq_queue *cfqq;unsigned int max_dispatch; +unsigned long delay;if (!cfqd->busy_queues)return 0 ;
@@ -1297,19 +1301,26 @@ static int cfq_dispatch_requests(struct/* * Drain async requests before we start sync IO */ -if (cfq_cfqq_idle_window (cfqq) && cfqd->rq_in_driver[BLK_RW_ASYNC ]) +if (cfq_cfqq_idle_window (cfqq) && cfqd->rq_in_driver[BLK_RW_ASYNC ]) { + cfqd->desktop_dispatch_ts = jiffies;return 0 ; + }/* * If this is an async queue and we have sync IO in flight, let it wait */ -if (cfqd->sync_flight && !cfq_cfqq_sync (cfqq)) +if (cfqd->sync_flight && !cfq_cfqq_sync (cfqq)) { + cfqd->desktop_dispatch_ts = jiffies;return 0 ; + } max_dispatch = cfqd->cfq_quantum;if (cfq_class_idle (cfqq)) max_dispatch =1 ; +if (cfqd->busy_queues >1 ) + cfqd->desktop_dispatch_ts = jiffies; +/* * Does this cfqq already have too much IO in flight? */
@@ -1327,6 +1338,16 @@ static int cfq_dispatch_requests(structreturn 0 ;/* +* Don't start overloading until we've been alone for a bit. +*/ +if (cfqd->cfq_desktop_dispatch) { + delay = cfqd->desktop_dispatch_ts + cfq_slice_sync; + +if (time_before (jiffies, max_delay)) +return 0 ; + } + +/* * we are the only queue, allow up to 4 times of 'quantum' */ if (cfqq->dispatched >=4 * max_dispatch)
@@ -1942,7 +1963,7 @@ static void cfq_update_idle_window(struct cfq_data *cfqd, struct cfq_queue *cfqq, struct cfq_io_context *cic) { - int old_idle, enable_idle; + int old_idle, enable_idle, seeky = 0;/* * Don't idle for async or idle io prio class
@@ -1950,10 +1971,20 @@ cfq_update_idle_window(struct cfq_data *if (!cfq_cfqq_sync (cfqq) ||cfq_class_idle (cfqq))return ; +if (cfqd->hw_tag) { +if (CIC_SEEKY (cic)) + seeky =1 ; +/* +* If seeky or incalculable seekiness, delay overloading. +*/ +if (seeky || !sample_valid (cic->seek_samples)) + cfqd->desktop_dispatch_ts = jiffies; + } + enable_idle = old_idle =cfq_cfqq_idle_window (cfqq);if (!atomic_read (&cic->ioc->nr_tasks) || !cfqd->cfq_slice_idle || - (!cfqd->cfq_desktop && cfqd->hw_tag &&CIC_SEEKY (cic))) + (!cfqd->cfq_desktop && seeky)) enable_idle =0 ;else if (sample_valid (cic->ttime_samples)) {if (cic->ttime_mean > cfqd->cfq_slice_idle)
@@ -2483,6 +2514,9 @@ static void *cfq_init_queue(struct reque cfqd->cfq_slice_async_rq = cfq_slice_async_rq; cfqd->cfq_slice_idle = cfq_slice_idle; cfqd->cfq_desktop =1 ; + cfqd->cfq_desktop_dispatch =1 ; + + cfqd->desktop_dispatch_ts =INITIAL_JIFFIES ; cfqd->hw_tag =1 ;return cfqd;
@@ -2553,6 +2587,7 @@ SHOW_FUNCTION(cfq_slice_sync_show, cfqd-SHOW_FUNCTION (cfq_slice_async_show, cfqd->cfq_slice[0 ],1 );SHOW_FUNCTION (cfq_slice_async_rq_show, cfqd->cfq_slice_async_rq,0 );SHOW_FUNCTION (cfq_desktop_show, cfqd->cfq_desktop,0 ); +SHOW_FUNCTION (cfq_desktop_dispatch_show, cfqd->cfq_desktop_dispatch,0 );#undef SHOW_FUNCTION#define STORE_FUNCTION (__FUNC, __PTR,MIN ,MAX , __CONV) \
@@ -2585,6 +2620,7 @@ STORE_FUNCTION(cfq_slice_async_store, &cSTORE_FUNCTION (cfq_slice_async_rq_store, &cfqd->cfq_slice_async_rq,1 ,UINT_MAX ,0 );STORE_FUNCTION (cfq_desktop_store, &cfqd->cfq_desktop,0 ,1 ,0 ); +STORE_FUNCTION (cfq_desktop_dispatch_store, &cfqd->cfq_desktop_dispatch,0 ,1 ,0 );#undef STORE_FUNCTION#define CFQ_ATTR (name) \
@@ -2601,6 +2637,7 @@ static struct elv_fs_entry cfq_attrs[] =CFQ_ATTR (slice_async_rq),CFQ_ATTR (slice_idle),CFQ_ATTR (desktop), +CFQ_ATTR (desktop_dispatch), __ATTR_NULL };
tweaking IO latency [was Re: IO scheduler based IO controller V10]
P.S. now may be a good time to finally exit thread (and maybe trim cc?)
Quoted 169 lines · Mike Galbraith
Quoted 8 lines · Mike Galbraith
Quoted 6 lines · Jens Axboe
If you could do a cleaned up version of your overload patch based on this:
http://git.kernel.dk/?p=linux-2.6-block.git;a=commit;h=1d2235152dc745c6d94bedb550fea84cffdbf768
then lets take it from there.
Note to self: build the darn thing after last minute changes.
Block: Delay overloading of CFQ queues to improve read latency.
Introduce a delay maximum dispatch timestamp, and stamp it when:
1. we encounter a known seeky or possibly new sync IO queue.
2. the current queue may go idle and we're draining async IO.
3. we have sync IO in flight and are servicing an async queue. 4 we are not the sole user of disk.
Disallow exceeding quantum if any of these events have occurred recently.Protect this behavioral change with a "
desktop_dispatch" knob and default it to "on".. providing an easy means of regression verification prior to hate-mail dispatch :) to CC list.Signed-off-by Mike Galbraith <[email protected]>Cc Jens Axboe <[email protected]>Cc Linus Torvalds <[email protected]>Cc Andrew Morton <[email protected]>... others who let somewhat hacky tweak slip by
--- block/cfq-iosched.c | 45 +++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 41 insertions(+), 4 deletions(-)Index:
linux-2.6/block/cfq-iosched.c
===================================================================block/cfq-iosched.c +41
-4@@ -174,6 +174,9 @@ struct cfq_data {unsigned int cfq_slice_async_rq;unsigned int cfq_slice_idle;unsigned int cfq_desktop; +unsigned int cfq_desktop_dispatch; + +unsigned long desktop_dispatch_ts;struct list_head cic_list;@@ -1283,6 +1286,7 @@ static int cfq_dispatch_requests(structstruct cfq_data *cfqd = q->elevator->elevator_data;struct cfq_queue *cfqq;unsigned int max_dispatch; +unsigned long delay;if (!cfqd->busy_queues)return 0 ;@@ -1297,19 +1301,26 @@ static int cfq_dispatch_requests(struct/* * Drain async requests before we start sync IO */ -if (cfq_cfqq_idle_window (cfqq) && cfqd->rq_in_driver[BLK_RW_ASYNC ]) +if (cfq_cfqq_idle_window (cfqq) && cfqd->rq_in_driver[BLK_RW_ASYNC ]) { + cfqd->desktop_dispatch_ts = jiffies;return 0 ; + }/* * If this is an async queue and we have sync IO in flight, let it wait */ -if (cfqd->sync_flight && !cfq_cfqq_sync (cfqq)) +if (cfqd->sync_flight && !cfq_cfqq_sync (cfqq)) { + cfqd->desktop_dispatch_ts = jiffies;return 0 ; + } max_dispatch = cfqd->cfq_quantum;if (cfq_class_idle (cfqq)) max_dispatch =1 ; +if (cfqd->busy_queues >1 ) + cfqd->desktop_dispatch_ts = jiffies; +/* * Does this cfqq already have too much IO in flight? */ @@ -1327,6 +1338,16 @@ static int cfq_dispatch_requests(structreturn 0 ;/* +* Don't start overloading until we've been alone for a bit. +*/ +if (cfqd->cfq_desktop_dispatch) { + delay = cfqd->desktop_dispatch_ts + cfq_slice_sync; + +if (time_before (jiffies, max_delay)) +return 0 ; + } + +/* * we are the only queue, allow up to 4 times of 'quantum' */ if (cfqq->dispatched >=4 * max_dispatch)@@ -1942,7 +1963,7 @@ static void cfq_update_idle_window(struct cfq_data *cfqd, struct cfq_queue *cfqq, struct cfq_io_context *cic) { - int old_idle, enable_idle; + int old_idle, enable_idle, seeky = 0;/* * Don't idle for async or idle io prio class @@ -1950,10 +1971,20 @@ cfq_update_idle_window(struct cfq_data *if (!cfq_cfqq_sync (cfqq) ||cfq_class_idle (cfqq))return ; +if (cfqd->hw_tag) { +if (CIC_SEEKY (cic)) + seeky =1 ; +/* +* If seeky or incalculable seekiness, delay overloading. +*/ +if (seeky || !sample_valid (cic->seek_samples)) + cfqd->desktop_dispatch_ts = jiffies; + } + enable_idle = old_idle =cfq_cfqq_idle_window (cfqq);if (!atomic_read (&cic->ioc->nr_tasks) || !cfqd->cfq_slice_idle || - (!cfqd->cfq_desktop && cfqd->hw_tag &&CIC_SEEKY (cic))) + (!cfqd->cfq_desktop && seeky)) enable_idle =0 ;else if (sample_valid (cic->ttime_samples)) {if (cic->ttime_mean > cfqd->cfq_slice_idle)@@ -2483,6 +2514,9 @@ static void *cfq_init_queue(struct reque cfqd->cfq_slice_async_rq = cfq_slice_async_rq; cfqd->cfq_slice_idle = cfq_slice_idle; cfqd->cfq_desktop =1 ; + cfqd->cfq_desktop_dispatch =1 ; + + cfqd->desktop_dispatch_ts =INITIAL_JIFFIES ; cfqd->hw_tag =1 ;return cfqd;@@ -2553,6 +2587,7 @@ SHOW_FUNCTION(cfq_slice_sync_show, cfqd-SHOW_FUNCTION (cfq_slice_async_show, cfqd->cfq_slice[0 ],1 );SHOW_FUNCTION (cfq_slice_async_rq_show, cfqd->cfq_slice_async_rq,0 );SHOW_FUNCTION (cfq_desktop_show, cfqd->cfq_desktop,0 ); +SHOW_FUNCTION (cfq_desktop_dispatch_show, cfqd->cfq_desktop_dispatch,0 );#undef SHOW_FUNCTION#define STORE_FUNCTION (__FUNC, __PTR,MIN ,MAX , __CONV) \@@ -2585,6 +2620,7 @@ STORE_FUNCTION(cfq_slice_async_store, &cSTORE_FUNCTION (cfq_slice_async_rq_store, &cfqd->cfq_slice_async_rq,1 ,UINT_MAX ,0 );STORE_FUNCTION (cfq_desktop_store, &cfqd->cfq_desktop,0 ,1 ,0 ); +STORE_FUNCTION (cfq_desktop_dispatch_store, &cfqd->cfq_desktop_dispatch,0 ,1 ,0 );#undef STORE_FUNCTION#define CFQ_ATTR (name) \@@ -2601,6 +2637,7 @@ static struct elv_fs_entry cfq_attrs[] =CFQ_ATTR (slice_async_rq),CFQ_ATTR (slice_idle),CFQ_ATTR (desktop), +CFQ_ATTR (desktop_dispatch), __ATTR_NULL };
A message that is not in the archive (sent off-list or not yet received): [email protected]
tweaking IO latency [was Re: IO scheduler based IO controller V10]
P.S. now may be a good time to finally exit thread (and maybe trim cc?)
Quoted 169 lines · Mike Galbraith
Quoted 8 lines · Mike Galbraith
Quoted 6 lines · Jens Axboe
If you could do a cleaned up version of your overload patch based on this:
http://git.kernel.dk/?p=linux-2.6-block.git;a=commit;h=1d2235152dc745c6d94bedb550fea84cffdbf768
then lets take it from there.
Note to self: build the darn thing after last minute changes.
Block: Delay overloading of CFQ queues to improve read latency.
Introduce a delay maximum dispatch timestamp, and stamp it when:
1. we encounter a known seeky or possibly new sync IO queue.
2. the current queue may go idle and we're draining async IO.
3. we have sync IO in flight and are servicing an async queue. 4 we are not the sole user of disk.
Disallow exceeding quantum if any of these events have occurred recently.Protect this behavioral change with a "
desktop_dispatch" knob and default it to "on".. providing an easy means of regression verification prior to hate-mail dispatch :) to CC list.Signed-off-by Mike Galbraith <[email protected]>Cc Jens Axboe <[email protected]>Cc Linus Torvalds <torvalds-de/[email protected]>Cc Andrew Morton <akpm-de/[email protected]>... others who let somewhat hacky tweak slip by
--- block/cfq-iosched.c | 45 +++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 41 insertions(+), 4 deletions(-)Index:
linux-2.6/block/cfq-iosched.c
===================================================================block/cfq-iosched.c +41
-4@@ -174,6 +174,9 @@ struct cfq_data {unsigned int cfq_slice_async_rq;unsigned int cfq_slice_idle;unsigned int cfq_desktop; +unsigned int cfq_desktop_dispatch; + +unsigned long desktop_dispatch_ts;struct list_head cic_list;@@ -1283,6 +1286,7 @@ static int cfq_dispatch_requests(structstruct cfq_data *cfqd = q->elevator->elevator_data;struct cfq_queue *cfqq;unsigned int max_dispatch; +unsigned long delay;if (!cfqd->busy_queues)return 0 ;@@ -1297,19 +1301,26 @@ static int cfq_dispatch_requests(struct/* * Drain async requests before we start sync IO */ -if (cfq_cfqq_idle_window (cfqq) && cfqd->rq_in_driver[BLK_RW_ASYNC ]) +if (cfq_cfqq_idle_window (cfqq) && cfqd->rq_in_driver[BLK_RW_ASYNC ]) { + cfqd->desktop_dispatch_ts = jiffies;return 0 ; + }/* * If this is an async queue and we have sync IO in flight, let it wait */ -if (cfqd->sync_flight && !cfq_cfqq_sync (cfqq)) +if (cfqd->sync_flight && !cfq_cfqq_sync (cfqq)) { + cfqd->desktop_dispatch_ts = jiffies;return 0 ; + } max_dispatch = cfqd->cfq_quantum;if (cfq_class_idle (cfqq)) max_dispatch =1 ; +if (cfqd->busy_queues >1 ) + cfqd->desktop_dispatch_ts = jiffies; +/* * Does this cfqq already have too much IO in flight? */ @@ -1327,6 +1338,16 @@ static int cfq_dispatch_requests(structreturn 0 ;/* +* Don't start overloading until we've been alone for a bit. +*/ +if (cfqd->cfq_desktop_dispatch) { + delay = cfqd->desktop_dispatch_ts + cfq_slice_sync; + +if (time_before (jiffies, max_delay)) +return 0 ; + } + +/* * we are the only queue, allow up to 4 times of 'quantum' */ if (cfqq->dispatched >=4 * max_dispatch)@@ -1942,7 +1963,7 @@ static void cfq_update_idle_window(struct cfq_data *cfqd, struct cfq_queue *cfqq, struct cfq_io_context *cic) { - int old_idle, enable_idle; + int old_idle, enable_idle, seeky = 0;/* * Don't idle for async or idle io prio class @@ -1950,10 +1971,20 @@ cfq_update_idle_window(struct cfq_data *if (!cfq_cfqq_sync (cfqq) ||cfq_class_idle (cfqq))return ; +if (cfqd->hw_tag) { +if (CIC_SEEKY (cic)) + seeky =1 ; +/* +* If seeky or incalculable seekiness, delay overloading. +*/ +if (seeky || !sample_valid (cic->seek_samples)) + cfqd->desktop_dispatch_ts = jiffies; + } + enable_idle = old_idle =cfq_cfqq_idle_window (cfqq);if (!atomic_read (&cic->ioc->nr_tasks) || !cfqd->cfq_slice_idle || - (!cfqd->cfq_desktop && cfqd->hw_tag &&CIC_SEEKY (cic))) + (!cfqd->cfq_desktop && seeky)) enable_idle =0 ;else if (sample_valid (cic->ttime_samples)) {if (cic->ttime_mean > cfqd->cfq_slice_idle)@@ -2483,6 +2514,9 @@ static void *cfq_init_queue(struct reque cfqd->cfq_slice_async_rq = cfq_slice_async_rq; cfqd->cfq_slice_idle = cfq_slice_idle; cfqd->cfq_desktop =1 ; + cfqd->cfq_desktop_dispatch =1 ; + + cfqd->desktop_dispatch_ts =INITIAL_JIFFIES ; cfqd->hw_tag =1 ;return cfqd;@@ -2553,6 +2587,7 @@ SHOW_FUNCTION(cfq_slice_sync_show, cfqd-SHOW_FUNCTION (cfq_slice_async_show, cfqd->cfq_slice[0 ],1 );SHOW_FUNCTION (cfq_slice_async_rq_show, cfqd->cfq_slice_async_rq,0 );SHOW_FUNCTION (cfq_desktop_show, cfqd->cfq_desktop,0 ); +SHOW_FUNCTION (cfq_desktop_dispatch_show, cfqd->cfq_desktop_dispatch,0 );#undef SHOW_FUNCTION#define STORE_FUNCTION (__FUNC, __PTR,MIN ,MAX , __CONV) \@@ -2585,6 +2620,7 @@ STORE_FUNCTION(cfq_slice_async_store, &cSTORE_FUNCTION (cfq_slice_async_rq_store, &cfqd->cfq_slice_async_rq,1 ,UINT_MAX ,0 );STORE_FUNCTION (cfq_desktop_store, &cfqd->cfq_desktop,0 ,1 ,0 ); +STORE_FUNCTION (cfq_desktop_dispatch_store, &cfqd->cfq_desktop_dispatch,0 ,1 ,0 );#undef STORE_FUNCTION#define CFQ_ATTR (name) \@@ -2601,6 +2637,7 @@ static struct elv_fs_entry cfq_attrs[] =CFQ_ATTR (slice_async_rq),CFQ_ATTR (slice_idle),CFQ_ATTR (desktop), +CFQ_ATTR (desktop_dispatch), __ATTR_NULL };
Quoted 21 lines · Mike Galbraith
Quoted 8 lines · Mike Galbraith
Quoted 6 lines · Jens Axboe
If you could do a cleaned up version of your overload patch based on this:
http://git.kernel.dk/?p=linux-2.6-block.git;a=commit;h=1d2235152dc745c6d94bedb550fea84cffdbf768
then lets take it from there.
Note to self: build the darn thing after last minute changes.
Block: Delay overloading of CFQ queues to improve read latency.
Introduce a delay maximum dispatch timestamp, and stamp it when:
1. we encounter a known seeky or possibly new sync IO queue.
2. the current queue may go idle and we're draining async IO.
3. we have sync IO in flight and are servicing an async queue. 4 we are not the sole user of disk.
Disallow exceeding quantum if any of these events have occurred recently.
Quoted 3 lines
Protect this behavioral change with a "
desktop_dispatch" knob and default it to "on".. providing an easy means of regression verification prior to hate-mail dispatch :) to CC list.
It still doesn't build:
block/cfq-iosched.c: In function ?cfq_dispatch_requests?:block/cfq-iosched.c:1345: error: ?max_delay? undeclared (first use in this function)
After shutting down the computer yesterday, I was thinking a bit about this issue and how to solve it without incurring too much delay. If we add a stricter control of the depth, that may help. So instead of allowing up to max_quantum (or larger) depths, only allow gradual build up of that the farther we get away from a dispatch from the sync IO queues. For example, when switching to an async or seeky sync queue, initially allow just 1 in flight. For the next round, if there still hasn't been sync activity, allow 2, then 4, etc. If we see sync IO queue again, immediately drop to 1.
It could tie in with (or partly replace) the overload feature. The key to good latency and decent throughput is knowing when to allow queue build up and when not to.
Signature · 1 line
Jens Axboe
Quoted 13 lines · Mike Galbraith
Quoted 8 lines · Mike Galbraith
Quoted 6 lines · Jens Axboe
If you could do a cleaned up version of your overload patch based on this:
http://git.kernel.dk/?p=linux-2.6-block.git;a=commit;h=1d2235152dc745c6d94bedb550fea84cffdbf768
then lets take it from there.
Note to self: build the darn thing after last minute changes.
Quoted 9 lines
Block: Delay overloading of CFQ queues to improve read latency.
Introduce a delay maximum dispatch timestamp, and stamp it when:
1. we encounter a known seeky or possibly new sync IO queue.
2. the current queue may go idle and we're draining async IO.
3. we have sync IO in flight and are servicing an async queue. 4 we are not the sole user of disk.
Disallow exceeding quantum if any of these events have occurred recently.
So it looks like primarily the issue seems to be that we done lot of dispatch from async queue and if some sync queue comes in now, it will experience latencies.
For a ongoing seeky sync queue issue will be solved up to some extent because previously we did not choose to idle for that queue now we will idle, hence async queue will not get a chance to overload the dispatch queue.
For the sync queues where we choose not to enable idle, we still will see the latencies. Instead of time stamping on all the above events, can we just keep track of last sync request completed in the system and don't allow async queue to flood/overload the dispatch queue with-in certain time limit of that last sync request completion. This just gives a buffer period to that sync queue to come back and submit more requests and still not suffer large latencies?
Thanks
Vivek
Quoted 148 lines
Protect this behavioral change with a "
desktop_dispatch" knob and default it to "on".. providing an easy means of regression verification prior to hate-mail dispatch :) to CC list.Signed-off-by Mike Galbraith <[email protected]>Cc Jens Axboe <[email protected]>Cc Linus Torvalds <torvalds-de/[email protected]>Cc Andrew Morton <akpm-de/[email protected]>... others who let somewhat hacky tweak slip by
--- block/cfq-iosched.c | 45 +++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 41 insertions(+), 4 deletions(-)Index:
linux-2.6/block/cfq-iosched.c
===================================================================block/cfq-iosched.c +41
-4@@ -174,6 +174,9 @@ struct cfq_data {unsigned int cfq_slice_async_rq;unsigned int cfq_slice_idle;unsigned int cfq_desktop; +unsigned int cfq_desktop_dispatch; + +unsigned long desktop_dispatch_ts;struct list_head cic_list;@@ -1283,6 +1286,7 @@ static int cfq_dispatch_requests(structstruct cfq_data *cfqd = q->elevator->elevator_data;struct cfq_queue *cfqq;unsigned int max_dispatch; +unsigned long delay;if (!cfqd->busy_queues)return 0 ;@@ -1297,19 +1301,26 @@ static int cfq_dispatch_requests(struct/* * Drain async requests before we start sync IO */ -if (cfq_cfqq_idle_window (cfqq) && cfqd->rq_in_driver[BLK_RW_ASYNC ]) +if (cfq_cfqq_idle_window (cfqq) && cfqd->rq_in_driver[BLK_RW_ASYNC ]) { + cfqd->desktop_dispatch_ts = jiffies;return 0 ; + }/* * If this is an async queue and we have sync IO in flight, let it wait */ -if (cfqd->sync_flight && !cfq_cfqq_sync (cfqq)) +if (cfqd->sync_flight && !cfq_cfqq_sync (cfqq)) { + cfqd->desktop_dispatch_ts = jiffies;return 0 ; + } max_dispatch = cfqd->cfq_quantum;if (cfq_class_idle (cfqq)) max_dispatch =1 ; +if (cfqd->busy_queues >1 ) + cfqd->desktop_dispatch_ts = jiffies; +/* * Does this cfqq already have too much IO in flight? */ @@ -1327,6 +1338,16 @@ static int cfq_dispatch_requests(structreturn 0 ;/* +* Don't start overloading until we've been alone for a bit. +*/ +if (cfqd->cfq_desktop_dispatch) { + delay = cfqd->desktop_dispatch_ts + cfq_slice_sync; + +if (time_before (jiffies, max_delay)) +return 0 ; + } + +/* * we are the only queue, allow up to 4 times of 'quantum' */ if (cfqq->dispatched >=4 * max_dispatch)@@ -1942,7 +1963,7 @@ static void cfq_update_idle_window(struct cfq_data *cfqd, struct cfq_queue *cfqq, struct cfq_io_context *cic) { - int old_idle, enable_idle; + int old_idle, enable_idle, seeky = 0;/* * Don't idle for async or idle io prio class @@ -1950,10 +1971,20 @@ cfq_update_idle_window(struct cfq_data *if (!cfq_cfqq_sync (cfqq) ||cfq_class_idle (cfqq))return ; +if (cfqd->hw_tag) { +if (CIC_SEEKY (cic)) + seeky =1 ; +/* +* If seeky or incalculable seekiness, delay overloading. +*/ +if (seeky || !sample_valid (cic->seek_samples)) + cfqd->desktop_dispatch_ts = jiffies; + } + enable_idle = old_idle =cfq_cfqq_idle_window (cfqq);if (!atomic_read (&cic->ioc->nr_tasks) || !cfqd->cfq_slice_idle || - (!cfqd->cfq_desktop && cfqd->hw_tag &&CIC_SEEKY (cic))) + (!cfqd->cfq_desktop && seeky)) enable_idle =0 ;else if (sample_valid (cic->ttime_samples)) {if (cic->ttime_mean > cfqd->cfq_slice_idle)@@ -2483,6 +2514,9 @@ static void *cfq_init_queue(struct reque cfqd->cfq_slice_async_rq = cfq_slice_async_rq; cfqd->cfq_slice_idle = cfq_slice_idle; cfqd->cfq_desktop =1 ; + cfqd->cfq_desktop_dispatch =1 ; + + cfqd->desktop_dispatch_ts =INITIAL_JIFFIES ; cfqd->hw_tag =1 ;return cfqd;@@ -2553,6 +2587,7 @@ SHOW_FUNCTION(cfq_slice_sync_show, cfqd-SHOW_FUNCTION (cfq_slice_async_show, cfqd->cfq_slice[0 ],1 );SHOW_FUNCTION (cfq_slice_async_rq_show, cfqd->cfq_slice_async_rq,0 );SHOW_FUNCTION (cfq_desktop_show, cfqd->cfq_desktop,0 ); +SHOW_FUNCTION (cfq_desktop_dispatch_show, cfqd->cfq_desktop_dispatch,0 );#undef SHOW_FUNCTION#define STORE_FUNCTION (__FUNC, __PTR,MIN ,MAX , __CONV) \@@ -2585,6 +2620,7 @@ STORE_FUNCTION(cfq_slice_async_store, &cSTORE_FUNCTION (cfq_slice_async_rq_store, &cfqd->cfq_slice_async_rq,1 ,UINT_MAX ,0 );STORE_FUNCTION (cfq_desktop_store, &cfqd->cfq_desktop,0 ,1 ,0 ); +STORE_FUNCTION (cfq_desktop_dispatch_store, &cfqd->cfq_desktop_dispatch,0 ,1 ,0 );#undef STORE_FUNCTION#define CFQ_ATTR (name) \@@ -2601,6 +2637,7 @@ static struct elv_fs_entry cfq_attrs[] =CFQ_ATTR (slice_async_rq),CFQ_ATTR (slice_idle),CFQ_ATTR (desktop), +CFQ_ATTR (desktop_dispatch), __ATTR_NULL };
Quoted 21 lines · Mike Galbraith
Quoted 8 lines · Mike Galbraith
Quoted 6 lines · Jens Axboe
If you could do a cleaned up version of your overload patch based on this:
http://git.kernel.dk/?p=linux-2.6-block.git;a=commit;h=1d2235152dc745c6d94bedb550fea84cffdbf768
then lets take it from there.
Note to self: build the darn thing after last minute changes.
Block: Delay overloading of CFQ queues to improve read latency.
Introduce a delay maximum dispatch timestamp, and stamp it when:
1. we encounter a known seeky or possibly new sync IO queue.
2. the current queue may go idle and we're draining async IO.
3. we have sync IO in flight and are servicing an async queue. 4 we are not the sole user of disk.
Disallow exceeding quantum if any of these events have occurred recently.
Quoted 3 lines
Protect this behavioral change with a "
desktop_dispatch" knob and default it to "on".. providing an easy means of regression verification prior to hate-mail dispatch :) to CC list.
It still doesn't build:
block/cfq-iosched.c: In function ?cfq_dispatch_requests?:block/cfq-iosched.c:1345: error: ?max_delay? undeclared (first use in this function)
After shutting down the computer yesterday, I was thinking a bit about this issue and how to solve it without incurring too much delay. If we add a stricter control of the depth, that may help. So instead of allowing up to max_quantum (or larger) depths, only allow gradual build up of that the farther we get away from a dispatch from the sync IO queues. For example, when switching to an async or seeky sync queue, initially allow just 1 in flight. For the next round, if there still hasn't been sync activity, allow 2, then 4, etc. If we see sync IO queue again, immediately drop to 1.
It could tie in with (or partly replace) the overload feature. The key to good latency and decent throughput is knowing when to allow queue build up and when not to.
Signature · 1 line
Jens Axboe
Quoted 10 lines · Jens Axboe
After shutting down the computer yesterday, I was thinking a bit about this issue and how to solve it without incurring too much delay. If we add a stricter control of the depth, that may help. So instead of allowing up to
max_quantum(or larger) depths, only allow gradual build up of that the farther we get away from a dispatch from the sync IO queues. For example, when switching to an async or seeky sync queue, initially allow just 1 in flight. For the next round, if there still hasn't been sync activity, allow 2, then 4, etc. If we see sync IO queue again, immediately drop to 1.
Quoted 3 lines
It could tie in with (or partly replace) the overload feature. The key to good latency and decent throughput is knowing when to allow queue build up and when not to.
Hm. Starting at 1 sounds a bit thin (like IDLE), multiple iterations to build/unleash any sizable IO, but that's just my gut talking.
-Mike
Hi,
Quoted 12 lines · Mike Galbraith
Quoted 10 lines · Jens Axboe
After shutting down the computer yesterday, I was thinking a bit about this issue and how to solve it without incurring too much delay. If we add a stricter control of the depth, that may help. So instead of allowing up to
max_quantum(or larger) depths, only allow gradual build up of that the farther we get away from a dispatch from the sync IO queues. For example, when switching to an async or seeky sync queue, initially allow just 1 in flight. For the next round, if there still hasn't been sync activity, allow 2, then 4, etc. If we see sync IO queue again, immediately drop to 1.
I would limit just async I/O. Seeky sync queues are automatically throttled by being sync, and have already high latency, so we shouldn't increase it artificially. I think, instead, that we should send multiple seeky requests (possibly coming from different queues) at once. They will help especially with raid devices, where the seeks for requests going to different disks will happen in parallel.
Quoted 7 lines
Quoted 3 lines
It could tie in with (or partly replace) the overload feature. The key to good latency and decent throughput is knowing when to allow queue build up and when not to.
Hm. Starting at 1 sounds a bit thin (like IDLE), multiple iterations to build/unleash any sizable IO, but that's just my gut talking.
On the other hand, sending 1 write first and then waiting it to complete before submitting new ones, will help performing more merges, so the subsequent requests will be bigger and thus more efficient.
Corrado
Quoted 8 lines
-Mike
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Signature · 5 lines
__________________________________________________________________________ dott. Corrado Zoccolo mailto:[email protected] PhD - Department of Computer Science - University of Pisa, Italy --------------------------------------------------------------------------
Quoted 15 lines · Corrado Zoccolo
Hi,
Quoted 12 lines · Mike Galbraith
Quoted 10 lines · Jens Axboe
After shutting down the computer yesterday, I was thinking a bit about this issue and how to solve it without incurring too much delay. If we add a stricter control of the depth, that may help. So instead of allowing up to
max_quantum(or larger) depths, only allow gradual build up of that the farther we get away from a dispatch from the sync IO queues. For example, when switching to an async or seeky sync queue, initially allow just 1 in flight. For the next round, if there still hasn't been sync activity, allow 2, then 4, etc. If we see sync IO queue again, immediately drop to 1.
Quoted 7 lines
I would limit just async I/O. Seeky sync queues are automatically throttled by being sync, and have already high latency, so we shouldn't increase it artificially. I think, instead, that we should send multiple seeky requests (possibly coming from different queues) at once. They will help especially with raid devices, where the seeks for requests going to different disks will happen in parallel.
Async is the prime offendor, definitely.
Quoted 10 lines
Quoted 7 lines
Quoted 3 lines
It could tie in with (or partly replace) the overload feature. The key to good latency and decent throughput is knowing when to allow queue build up and when not to.
Hm. Starting at 1 sounds a bit thin (like IDLE), multiple iterations to build/unleash any sizable IO, but that's just my gut talking.
On the other hand, sending 1 write first and then waiting it to complete before submitting new ones, will help performing more merges, so the subsequent requests will be bigger and thus more efficient.
Usually async writes stack up very quickly, so as long as you don't drain completely, the merging will happen automagically anyway.
Signature · 1 line
Jens Axboe
A message that is not in the archive (sent off-list or not yet received): 4e5e476b0910030212y50f97d97nc2e17c35d855cc63-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org
Quoted 15 lines · Corrado Zoccolo
Hi,
Quoted 12 lines · Mike Galbraith
Quoted 10 lines · Jens Axboe
After shutting down the computer yesterday, I was thinking a bit about this issue and how to solve it without incurring too much delay. If we add a stricter control of the depth, that may help. So instead of allowing up to
max_quantum(or larger) depths, only allow gradual build up of that the farther we get away from a dispatch from the sync IO queues. For example, when switching to an async or seeky sync queue, initially allow just 1 in flight. For the next round, if there still hasn't been sync activity, allow 2, then 4, etc. If we see sync IO queue again, immediately drop to 1.
Quoted 7 lines
I would limit just async I/O. Seeky sync queues are automatically throttled by being sync, and have already high latency, so we shouldn't increase it artificially. I think, instead, that we should send multiple seeky requests (possibly coming from different queues) at once. They will help especially with raid devices, where the seeks for requests going to different disks will happen in parallel.
Async is the prime offendor, definitely.
Quoted 10 lines
Quoted 7 lines
Quoted 3 lines
It could tie in with (or partly replace) the overload feature. The key to good latency and decent throughput is knowing when to allow queue build up and when not to.
Hm. Starting at 1 sounds a bit thin (like IDLE), multiple iterations to build/unleash any sizable IO, but that's just my gut talking.
On the other hand, sending 1 write first and then waiting it to complete before submitting new ones, will help performing more merges, so the subsequent requests will be bigger and thus more efficient.
Usually async writes stack up very quickly, so as long as you don't drain completely, the merging will happen automagically anyway.
Signature · 1 line
Jens Axboe
A message that is not in the archive (sent off-list or not yet received): 1254560434.17052.14.camel-YqMYhexLQo1vAv1Ojkdn7Q@public.gmane.org
Hi,
Quoted 12 lines · Mike Galbraith
Quoted 10 lines · Jens Axboe
After shutting down the computer yesterday, I was thinking a bit about this issue and how to solve it without incurring too much delay. If we add a stricter control of the depth, that may help. So instead of allowing up to
max_quantum(or larger) depths, only allow gradual build up of that the farther we get away from a dispatch from the sync IO queues. For example, when switching to an async or seeky sync queue, initially allow just 1 in flight. For the next round, if there still hasn't been sync activity, allow 2, then 4, etc. If we see sync IO queue again, immediately drop to 1.
I would limit just async I/O. Seeky sync queues are automatically throttled by being sync, and have already high latency, so we shouldn't increase it artificially. I think, instead, that we should send multiple seeky requests (possibly coming from different queues) at once. They will help especially with raid devices, where the seeks for requests going to different disks will happen in parallel.
Quoted 7 lines
Quoted 3 lines
It could tie in with (or partly replace) the overload feature. The key to good latency and decent throughput is knowing when to allow queue build up and when not to.
Hm. Starting at 1 sounds a bit thin (like IDLE), multiple iterations to build/unleash any sizable IO, but that's just my gut talking.
On the other hand, sending 1 write first and then waiting it to complete before submitting new ones, will help performing more merges, so the subsequent requests will be bigger and thus more efficient.
Corrado
Quoted 8 lines
-Mike
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Signature · 9 lines
__________________________________________________________________________ dott. Corrado Zoccolo mailto:[email protected] PhD - Department of Computer Science - University of Pisa, Italy -------------------------------------------------------------------------- _______________________________________________ Containers mailing list [email protected] https://lists.linux-foundation.org/mailman/listinfo/containers
Quoted 12 lines · Mike Galbraith
Quoted 10 lines · Jens Axboe
After shutting down the computer yesterday, I was thinking a bit about this issue and how to solve it without incurring too much delay. If we add a stricter control of the depth, that may help. So instead of allowing up to
max_quantum(or larger) depths, only allow gradual build up of that the farther we get away from a dispatch from the sync IO queues. For example, when switching to an async or seeky sync queue, initially allow just 1 in flight. For the next round, if there still hasn't been sync activity, allow 2, then 4, etc. If we see sync IO queue again, immediately drop to 1.
Quoted 6 lines
Quoted 3 lines
It could tie in with (or partly replace) the overload feature. The key to good latency and decent throughput is knowing when to allow queue build up and when not to.
Hm. Starting at 1 sounds a bit thin (like IDLE), multiple iterations to build/unleash any sizable IO, but that's just my gut talking.
Not sure, will need some testing of course. But it'll build up quickly.
Signature · 1 line
Jens Axboe
Quoted 12 lines · Mike Galbraith
Quoted 10 lines · Jens Axboe
After shutting down the computer yesterday, I was thinking a bit about this issue and how to solve it without incurring too much delay. If we add a stricter control of the depth, that may help. So instead of allowing up to
max_quantum(or larger) depths, only allow gradual build up of that the farther we get away from a dispatch from the sync IO queues. For example, when switching to an async or seeky sync queue, initially allow just 1 in flight. For the next round, if there still hasn't been sync activity, allow 2, then 4, etc. If we see sync IO queue again, immediately drop to 1.
Quoted 6 lines
Quoted 3 lines
It could tie in with (or partly replace) the overload feature. The key to good latency and decent throughput is knowing when to allow queue build up and when not to.
Hm. Starting at 1 sounds a bit thin (like IDLE), multiple iterations to build/unleash any sizable IO, but that's just my gut talking.
Not sure, will need some testing of course. But it'll build up quickly.
Signature · 1 line
Jens Axboe
A message that is not in the archive (sent off-list or not yet received): 20091003072401.GV31616-tSWWG44O7X1aa/[email protected]
Quoted 10 lines · Jens Axboe
After shutting down the computer yesterday, I was thinking a bit about this issue and how to solve it without incurring too much delay. If we add a stricter control of the depth, that may help. So instead of allowing up to
max_quantum(or larger) depths, only allow gradual build up of that the farther we get away from a dispatch from the sync IO queues. For example, when switching to an async or seeky sync queue, initially allow just 1 in flight. For the next round, if there still hasn't been sync activity, allow 2, then 4, etc. If we see sync IO queue again, immediately drop to 1.
Quoted 3 lines
It could tie in with (or partly replace) the overload feature. The key to good latency and decent throughput is knowing when to allow queue build up and when not to.
Hm. Starting at 1 sounds a bit thin (like IDLE), multiple iterations to build/unleash any sizable IO, but that's just my gut talking.
-Mike
Quoted 13 lines · Mike Galbraith
Quoted 8 lines · Mike Galbraith
Quoted 6 lines · Jens Axboe
If you could do a cleaned up version of your overload patch based on this:
http://git.kernel.dk/?p=linux-2.6-block.git;a=commit;h=1d2235152dc745c6d94bedb550fea84cffdbf768
then lets take it from there.
Note to self: build the darn thing after last minute changes.
Quoted 9 lines
Block: Delay overloading of CFQ queues to improve read latency.
Introduce a delay maximum dispatch timestamp, and stamp it when:
1. we encounter a known seeky or possibly new sync IO queue.
2. the current queue may go idle and we're draining async IO.
3. we have sync IO in flight and are servicing an async queue. 4 we are not the sole user of disk.
Disallow exceeding quantum if any of these events have occurred recently.
So it looks like primarily the issue seems to be that we done lot of dispatch from async queue and if some sync queue comes in now, it will experience latencies.
For a ongoing seeky sync queue issue will be solved up to some extent because previously we did not choose to idle for that queue now we will idle, hence async queue will not get a chance to overload the dispatch queue.
For the sync queues where we choose not to enable idle, we still will see the latencies. Instead of time stamping on all the above events, can we just keep track of last sync request completed in the system and don't allow async queue to flood/overload the dispatch queue with-in certain time limit of that last sync request completion. This just gives a buffer period to that sync queue to come back and submit more requests and still not suffer large latencies?
Thanks
Vivek
Quoted 148 lines
Protect this behavioral change with a "
desktop_dispatch" knob and default it to "on".. providing an easy means of regression verification prior to hate-mail dispatch :) to CC list.Signed-off-by Mike Galbraith <[email protected]>Cc Jens Axboe <[email protected]>Cc Linus Torvalds <[email protected]>Cc Andrew Morton <[email protected]>... others who let somewhat hacky tweak slip by
--- block/cfq-iosched.c | 45 +++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 41 insertions(+), 4 deletions(-)Index:
linux-2.6/block/cfq-iosched.c
===================================================================block/cfq-iosched.c +41
-4@@ -174,6 +174,9 @@ struct cfq_data {unsigned int cfq_slice_async_rq;unsigned int cfq_slice_idle;unsigned int cfq_desktop; +unsigned int cfq_desktop_dispatch; + +unsigned long desktop_dispatch_ts;struct list_head cic_list;@@ -1283,6 +1286,7 @@ static int cfq_dispatch_requests(structstruct cfq_data *cfqd = q->elevator->elevator_data;struct cfq_queue *cfqq;unsigned int max_dispatch; +unsigned long delay;if (!cfqd->busy_queues)return 0 ;@@ -1297,19 +1301,26 @@ static int cfq_dispatch_requests(struct/* * Drain async requests before we start sync IO */ -if (cfq_cfqq_idle_window (cfqq) && cfqd->rq_in_driver[BLK_RW_ASYNC ]) +if (cfq_cfqq_idle_window (cfqq) && cfqd->rq_in_driver[BLK_RW_ASYNC ]) { + cfqd->desktop_dispatch_ts = jiffies;return 0 ; + }/* * If this is an async queue and we have sync IO in flight, let it wait */ -if (cfqd->sync_flight && !cfq_cfqq_sync (cfqq)) +if (cfqd->sync_flight && !cfq_cfqq_sync (cfqq)) { + cfqd->desktop_dispatch_ts = jiffies;return 0 ; + } max_dispatch = cfqd->cfq_quantum;if (cfq_class_idle (cfqq)) max_dispatch =1 ; +if (cfqd->busy_queues >1 ) + cfqd->desktop_dispatch_ts = jiffies; +/* * Does this cfqq already have too much IO in flight? */ @@ -1327,6 +1338,16 @@ static int cfq_dispatch_requests(structreturn 0 ;/* +* Don't start overloading until we've been alone for a bit. +*/ +if (cfqd->cfq_desktop_dispatch) { + delay = cfqd->desktop_dispatch_ts + cfq_slice_sync; + +if (time_before (jiffies, max_delay)) +return 0 ; + } + +/* * we are the only queue, allow up to 4 times of 'quantum' */ if (cfqq->dispatched >=4 * max_dispatch)@@ -1942,7 +1963,7 @@ static void cfq_update_idle_window(struct cfq_data *cfqd, struct cfq_queue *cfqq, struct cfq_io_context *cic) { - int old_idle, enable_idle; + int old_idle, enable_idle, seeky = 0;/* * Don't idle for async or idle io prio class @@ -1950,10 +1971,20 @@ cfq_update_idle_window(struct cfq_data *if (!cfq_cfqq_sync (cfqq) ||cfq_class_idle (cfqq))return ; +if (cfqd->hw_tag) { +if (CIC_SEEKY (cic)) + seeky =1 ; +/* +* If seeky or incalculable seekiness, delay overloading. +*/ +if (seeky || !sample_valid (cic->seek_samples)) + cfqd->desktop_dispatch_ts = jiffies; + } + enable_idle = old_idle =cfq_cfqq_idle_window (cfqq);if (!atomic_read (&cic->ioc->nr_tasks) || !cfqd->cfq_slice_idle || - (!cfqd->cfq_desktop && cfqd->hw_tag &&CIC_SEEKY (cic))) + (!cfqd->cfq_desktop && seeky)) enable_idle =0 ;else if (sample_valid (cic->ttime_samples)) {if (cic->ttime_mean > cfqd->cfq_slice_idle)@@ -2483,6 +2514,9 @@ static void *cfq_init_queue(struct reque cfqd->cfq_slice_async_rq = cfq_slice_async_rq; cfqd->cfq_slice_idle = cfq_slice_idle; cfqd->cfq_desktop =1 ; + cfqd->cfq_desktop_dispatch =1 ; + + cfqd->desktop_dispatch_ts =INITIAL_JIFFIES ; cfqd->hw_tag =1 ;return cfqd;@@ -2553,6 +2587,7 @@ SHOW_FUNCTION(cfq_slice_sync_show, cfqd-SHOW_FUNCTION (cfq_slice_async_show, cfqd->cfq_slice[0 ],1 );SHOW_FUNCTION (cfq_slice_async_rq_show, cfqd->cfq_slice_async_rq,0 );SHOW_FUNCTION (cfq_desktop_show, cfqd->cfq_desktop,0 ); +SHOW_FUNCTION (cfq_desktop_dispatch_show, cfqd->cfq_desktop_dispatch,0 );#undef SHOW_FUNCTION#define STORE_FUNCTION (__FUNC, __PTR,MIN ,MAX , __CONV) \@@ -2585,6 +2620,7 @@ STORE_FUNCTION(cfq_slice_async_store, &cSTORE_FUNCTION (cfq_slice_async_rq_store, &cfqd->cfq_slice_async_rq,1 ,UINT_MAX ,0 );STORE_FUNCTION (cfq_desktop_store, &cfqd->cfq_desktop,0 ,1 ,0 ); +STORE_FUNCTION (cfq_desktop_dispatch_store, &cfqd->cfq_desktop_dispatch,0 ,1 ,0 );#undef STORE_FUNCTION#define CFQ_ATTR (name) \@@ -2601,6 +2637,7 @@ static struct elv_fs_entry cfq_attrs[] =CFQ_ATTR (slice_async_rq),CFQ_ATTR (slice_idle),CFQ_ATTR (desktop), +CFQ_ATTR (desktop_dispatch), __ATTR_NULL };
Do not overload dispatch queue (Was: Re: IO scheduler based IO controller V10)
Quoted 41 lines · Vivek Goyal
Quoted 13 lines · Mike Galbraith
Quoted 8 lines · Mike Galbraith
Quoted 6 lines · Jens Axboe
If you could do a cleaned up version of your overload patch based on this:
http://git.kernel.dk/?p=linux-2.6-block.git;a=commit;h=1d2235152dc745c6d94bedb550fea84cffdbf768
then lets take it from there.
Note to self: build the darn thing after last minute changes.
Quoted 9 lines
Block: Delay overloading of CFQ queues to improve read latency.
Introduce a delay maximum dispatch timestamp, and stamp it when:
1. we encounter a known seeky or possibly new sync IO queue.
2. the current queue may go idle and we're draining async IO.
3. we have sync IO in flight and are servicing an async queue. 4 we are not the sole user of disk.
Disallow exceeding quantum if any of these events have occurred recently.So it looks like primarily the issue seems to be that we done lot of dispatch from async queue and if some sync queue comes in now, it will experience latencies.
For a ongoing seeky sync queue issue will be solved up to some extent because previously we did not choose to idle for that queue now we will idle, hence async queue will not get a chance to overload the dispatch queue.
For the sync queues where we choose not to enable idle, we still will see the latencies. Instead of time stamping on all the above events, can we just keep track of last sync request completed in the system and don't allow async queue to flood/overload the dispatch queue with-in certain time limit of that last sync request completion. This just gives a buffer period to that sync queue to come back and submit more requests and still not suffer large latencies?
Quoted 3 lines
Thanks
Vivek
Hi Mike,
Following is a quick hack patch for the above idea. It is just compile and boot tested. Can you please see if it helps in your scenario.
Thanks
Vivek
o Do not allow more than max_dispatch requests from an async queue, if some sync request has finished recently. This is in the hope that sync activity is still going on in the system and we might receive a sync request soon. Most likely from a sync queue which finished a request and we did not enable idling on it.
--- block/cfq-iosched.c | 18 ++++++++++++++++-- 1 file changed, 16 insertions(+), 2 deletions(-)
block/cfq-iosched.c +16 -2
@@ -181,6 +181,8 @@ struct cfq_data {* Fallback dummy cfqq for extreme OOM conditions */ struct cfq_queue oom_cfqq; + +unsigned long last_end_sync_rq; };enum cfqq_state_flags {
@@ -1314,6 +1316,8 @@ static int cfq_dispatch_requests(struct* Does this cfqq already have too much IO in flight? */ if (cfqq->dispatched >= max_dispatch) { +unsigned long load_at = cfqd->last_end_sync_rq + cfq_slice_sync; +/* * idle queue must always only have a single IO in flight */
@@ -1327,6 +1331,14 @@ static int cfq_dispatch_requests(structreturn 0 ;/* +* If a sync request has completed recently, don't overload +* the dispatch queue yet with async requests. +*/ +if (cfqd->cfq_desktop && !cfq_cfqq_sync (cfqq) + &&time_before (jiffies, load_at)) +return 0 ; + +/* * we are the only queue, allow up to 4 times of 'quantum' */ if (cfqq->dispatched >=4 * max_dispatch)
@@ -2158,8 +2170,10 @@ static void cfq_completed_request(structif (cfq_cfqq_sync (cfqq)) cfqd->sync_flight--; -if (sync) +if (sync) {RQ_CIC (rq)->last_end_request = now; + cfqd->last_end_sync_rq = now; + }/* * If this is the active queue, check if it needs to be expired,
@@ -2483,7 +2497,7 @@ static void *cfq_init_queue(struct reque cfqd->cfq_slice_idle = cfq_slice_idle; cfqd->cfq_desktop =1 ; cfqd->hw_tag =1 ; - + cfqd->last_end_sync_rq = jiffies;return cfqd; }
Re: Do not overload dispatch queue (Was: Re: IO scheduler based IO controller V10)
Quoted 55 lines · Vivek Goyal
Quoted 41 lines · Vivek Goyal
Quoted 13 lines · Mike Galbraith
Quoted 8 lines · Mike Galbraith
Quoted 6 lines · Jens Axboe
If you could do a cleaned up version of your overload patch based on this:
http://git.kernel.dk/?p=linux-2.6-block.git;a=commit;h=1d2235152dc745c6d94bedb550fea84cffdbf768
then lets take it from there.
Note to self: build the darn thing after last minute changes.
Quoted 9 lines
Block: Delay overloading of CFQ queues to improve read latency.
Introduce a delay maximum dispatch timestamp, and stamp it when:
1. we encounter a known seeky or possibly new sync IO queue.
2. the current queue may go idle and we're draining async IO.
3. we have sync IO in flight and are servicing an async queue. 4 we are not the sole user of disk.
Disallow exceeding quantum if any of these events have occurred recently.So it looks like primarily the issue seems to be that we done lot of dispatch from async queue and if some sync queue comes in now, it will experience latencies.
For a ongoing seeky sync queue issue will be solved up to some extent because previously we did not choose to idle for that queue now we will idle, hence async queue will not get a chance to overload the dispatch queue.
For the sync queues where we choose not to enable idle, we still will see the latencies. Instead of time stamping on all the above events, can we just keep track of last sync request completed in the system and don't allow async queue to flood/overload the dispatch queue with-in certain time limit of that last sync request completion. This just gives a buffer period to that sync queue to come back and submit more requests and still not suffer large latencies?
Quoted 3 lines
Thanks
VivekHi Mike,
Following is a quick hack patch for the above idea. It is just compile and boot tested. Can you please see if it helps in your scenario.
Thanks
Vivek
Quoted 5 lines
o Do not allow more than
max_dispatchrequests from an async queue, if some sync request has finished recently. This is in the hope that sync activity is still going on in the system and we might receive a sync request soon. Most likely from a sync queue which finished a request and we did not enable idling on it.
This is pretty much identical to the scheme I described, except for the ramping of queue depth. I've applied it, it's nice and simple and I believe this will get rid of the worst of the problem.
Things probably end up being a bit simplistic, but we can always tweak around later.
Signature · 1 line
Jens Axboe
Re: Do not overload dispatch queue (Was: Re: IO scheduler based IO controller V10)
Quoted 62 lines · Jens Axboe
Quoted 55 lines · Vivek Goyal
Quoted 41 lines · Vivek Goyal
Quoted 13 lines · Mike Galbraith
Quoted 8 lines · Mike Galbraith
Quoted 6 lines · Jens Axboe
If you could do a cleaned up version of your overload patch based on this:
http://git.kernel.dk/?p=linux-2.6-block.git;a=commit;h=1d2235152dc745c6d94bedb550fea84cffdbf768
then lets take it from there.
Note to self: build the darn thing after last minute changes.
Quoted 9 lines
Block: Delay overloading of CFQ queues to improve read latency.
Introduce a delay maximum dispatch timestamp, and stamp it when:
1. we encounter a known seeky or possibly new sync IO queue.
2. the current queue may go idle and we're draining async IO.
3. we have sync IO in flight and are servicing an async queue. 4 we are not the sole user of disk.
Disallow exceeding quantum if any of these events have occurred recently.So it looks like primarily the issue seems to be that we done lot of dispatch from async queue and if some sync queue comes in now, it will experience latencies.
For a ongoing seeky sync queue issue will be solved up to some extent because previously we did not choose to idle for that queue now we will idle, hence async queue will not get a chance to overload the dispatch queue.
For the sync queues where we choose not to enable idle, we still will see the latencies. Instead of time stamping on all the above events, can we just keep track of last sync request completed in the system and don't allow async queue to flood/overload the dispatch queue with-in certain time limit of that last sync request completion. This just gives a buffer period to that sync queue to come back and submit more requests and still not suffer large latencies?
Quoted 3 lines
Thanks
VivekHi Mike,
Following is a quick hack patch for the above idea. It is just compile and boot tested. Can you please see if it helps in your scenario.
Thanks
VivekQuoted 5 lines
o Do not allow more than
max_dispatchrequests from an async queue, if some sync request has finished recently. This is in the hope that sync activity is still going on in the system and we might receive a sync request soon. Most likely from a sync queue which finished a request and we did not enable idling on it.
Quoted 6 lines
This is pretty much identical to the scheme I described, except for the ramping of queue depth. I've applied it, it's nice and simple and I believe this will get rid of the worst of the problem.
Things probably end up being a bit simplistic, but we can always tweak around later.
I have kept the overload delay period as "cfq_slice_sync" same as Mike had done. We shall have to experiment what is a good waiting perioed. Is 100ms too long if we are waiting for a request from same process which recently finished IO and we did not enable idle on it.
I guess we can tweak the delay period as we move along.
Thanks
Vivek
Re: Do not overload dispatch queue (Was: Re: IO scheduler based IO controller V10)
Quoted 6 lines · Vivek Goyal
I have kept the overload delay period as "
cfq_slice_sync" same as Mike had done. We shall have to experiment what is a good waiting perioed. Is 100ms too long if we are waiting for a request from same process which recently finished IO and we did not enable idle on it.I guess we can tweak the delay period as we move along.
I kept the delay period very short to minimize possible damage. Without the idle thing, it wasn't enough, but with, worked a treat, as does your patch.
-Mike
Re: Do not overload dispatch queue (Was: Re: IO scheduler based IO controller V10)
Quoted 7 lines · Mike Galbraith
Quoted 5 lines · Vivek Goyal
I have kept the overload delay period as "
cfq_slice_sync" same as Mike had done. We shall have to experiment what is a good waiting perioed. Is 100ms too long if we are waiting for a request from same process which recently finished IO and we did not enable idle on it.
Quoted 5 lines
Quoted 1 line
I guess we can tweak the delay period as we move along.
I kept the delay period very short to minimize possible damage. Without the idle thing, it wasn't enough, but with, worked a treat, as does your patch.
Can you test the current line up of patches in for-linus? It has the ramp up I talked about included as well.
Signature · 1 line
Jens Axboe
Re: Do not overload dispatch queue (Was: Re: IO scheduler based IO controller V10)
Quoted 8 lines · Jens Axboe
Quoted 7 lines · Mike Galbraith
Quoted 5 lines · Vivek Goyal
I have kept the overload delay period as "
cfq_slice_sync" same as Mike had done. We shall have to experiment what is a good waiting perioed. Is 100ms too long if we are waiting for a request from same process which recently finished IO and we did not enable idle on it.
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
I guess we can tweak the delay period as we move along.
I kept the delay period very short to minimize possible damage. Without the idle thing, it wasn't enough, but with, worked a treat, as does your patch.
Can you test the current line up of patches in for-linus? It has the ramp up I talked about included as well.
Sure. I'll go find it.
-Mike
A message that is not in the archive (sent off-list or not yet received): 20091003142840.GE31616-tSWWG44O7X1aa/[email protected]
Re: Do not overload dispatch queue (Was: Re: IO scheduler based IO controller V10)
Quoted 8 lines · Jens Axboe
Quoted 7 lines · Mike Galbraith
Quoted 5 lines · Vivek Goyal
I have kept the overload delay period as "
cfq_slice_sync" same as Mike had done. We shall have to experiment what is a good waiting perioed. Is 100ms too long if we are waiting for a request from same process which recently finished IO and we did not enable idle on it.
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
I guess we can tweak the delay period as we move along.
I kept the delay period very short to minimize possible damage. Without the idle thing, it wasn't enough, but with, worked a treat, as does your patch.
Can you test the current line up of patches in for-linus? It has the ramp up I talked about included as well.
Sure. I'll go find it.
-Mike