IO scheduler based IO controller V6
Subsystems (19)
Re: [PATCH 21/25] io-controller: Per cgroup request descriptor support
In reply to a message not in the archiveQuoted 58 lines · Vivek Goyal
Quoted 45 lines · Nauman Rafique
Quoted 37 lines · Gui Jianfeng
Quoted 28 lines · Vivek Goyal
o Currently a request queue has got fixed number of request descriptors for sync and async requests. Once the request descriptors are consumed, new processes are put to sleep and they effectively become serialized. Because sync and async queues are separate, async requests don't impact sync ones but if one is looking for fairness between async requests, that is not achievable if request queue descriptors become bottleneck.
o Make request descriptor's per io group so that if there is lots of IO going on in one cgroup, it does not impact the IO of other group.
o This is just one relatively simple way of doing things. This patch will probably change after the feedback. Folks have raised concerns that in hierchical setup, child's request descriptors should be capped by parent's request descriptors. May be we need to have per cgroup per device files in cgroups where one can specify the upper limit of request descriptors and whenever a cgroup is created one needs to assign request descritor limit making sure total sum of child's request descriptor is not more than of parent.
I guess something like memory controller. Anyway, that would be the next step. For the time being, we have implemented something simpler as follows.
o This patch implements the per cgroup request descriptors. request pool per queue is still common but every group will have its own wait list and its own count of request descriptors allocated to that group for sync and async queues. So effectively
request_listbecomes per io group property and not a global request queue feature.Quoted 7 lines
o Currently one can define
q->nr_requeststo limit request descriptors allocated for the queue. Now there is another tunableq->nr_group_requestswhich controls the requests descriptr limit per group.q->nr_requestssupercedesq->nr_group_requeststo make sure if there are lots of groups present, we don't end up allocating too many request descriptors on the queue.Quoted 5 lines
Hi Vivek,
In order to prevent
q->nr_requestsfrom becoming the bottle-neck of allocating requests, whether we can updatenr_requestsaccordingly when allocating or removing a cgroup?Quoted 6 lines
Vivek,
I agree with Gui here. In fact, it does not make much sense to keep thenr_requestslimit if we already have per cgroup limit in place. This change also simplifies code quite a bit, as we can get rid of all thatsleep_on_globallogic.Hi Nauman, Gui,
There were few reasons to keep a total limit on number of request descriptors (
q->nr_requests) apart from per group limit.
Quoted 8 lines
- We have this notion of queue being congested or not depending on out of
q->nr_requestshow many are currently being used. Writeback threads, some filesystems and other places make use of this information to either not to block or to avoid pushing too much of data on device if queue is congested.With
q->nr_requestsremoved, how do you define queue full and congested semantics?
We can still keep q->nr_requests around, but don't use that number to deny request descriptor allocation; only use it for defining queue full and congested semantics.
Quoted 36 lines
- I think
slee_on_globallogic makes sense even withoutq->nr_requests. Assume that a group allows request descriptor allocation but due to lack of memory, allocation fails. Where do you make this process wait to attempt next time? Making all such failed processes on gloabl list on queue instead of per group list makes more sense to me for following reasons.- If this is the first request allocation from the group and we make the process sleep on group list, it will never be woken up as no request from that group will complete.
- If there are many processes who failed request descriptor allocation, when some request completes, I think it is more fair to wake these up in FIFO manner to try out allocation again instead of waiting for request to complete from the group process belongs to. The reason being that io controller did not fail the request descriptor allocation.
So even if you get rid of
q->nr_requests, you still shall have to have some logic of global wait list where failed allocations can wait.- It is backward compatible and there are less chances of higher layers being broken due to this.
Gui, I think automatic updation of
q->nr_requestsis probably not a very good thing. It is user defined tunable and user does not expect this to change automatically.At this point of time I really can't think of simpler and cleaner way. Ideas are welcome.
Thanks
Vivek
A message that is not in the archive (sent off-list or not yet received): e98e18940907202255y5c7c546ei95d87e5a451ad0c2-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org
Re: [PATCH 21/25] io-controller: Per cgroup request descriptor support
Quoted 45 lines · Nauman Rafique
Quoted 37 lines · Gui Jianfeng
Quoted 28 lines · Vivek Goyal
o Currently a request queue has got fixed number of request descriptors for sync and async requests. Once the request descriptors are consumed, new processes are put to sleep and they effectively become serialized. Because sync and async queues are separate, async requests don't impact sync ones but if one is looking for fairness between async requests, that is not achievable if request queue descriptors become bottleneck.
o Make request descriptor's per io group so that if there is lots of IO going on in one cgroup, it does not impact the IO of other group.
o This is just one relatively simple way of doing things. This patch will probably change after the feedback. Folks have raised concerns that in hierchical setup, child's request descriptors should be capped by parent's request descriptors. May be we need to have per cgroup per device files in cgroups where one can specify the upper limit of request descriptors and whenever a cgroup is created one needs to assign request descritor limit making sure total sum of child's request descriptor is not more than of parent.
I guess something like memory controller. Anyway, that would be the next step. For the time being, we have implemented something simpler as follows.
o This patch implements the per cgroup request descriptors. request pool per queue is still common but every group will have its own wait list and its own count of request descriptors allocated to that group for sync and async queues. So effectively
request_listbecomes per io group property and not a global request queue feature.Quoted 7 lines
o Currently one can define
q->nr_requeststo limit request descriptors allocated for the queue. Now there is another tunableq->nr_group_requestswhich controls the requests descriptr limit per group.q->nr_requestssupercedesq->nr_group_requeststo make sure if there are lots of groups present, we don't end up allocating too many request descriptors on the queue.Quoted 5 lines
Hi Vivek,
In order to prevent
q->nr_requestsfrom becoming the bottle-neck of allocating requests, whether we can updatenr_requestsaccordingly when allocating or removing a cgroup?
Quoted 6 lines
Vivek,
I agree with Gui here. In fact, it does not make much sense to keep thenr_requestslimit if we already have per cgroup limit in place. This change also simplifies code quite a bit, as we can get rid of all thatsleep_on_globallogic.
Hi Nauman, Gui,
There were few reasons to keep a total limit on number of request descriptors (q->nr_requests) apart from per group limit.
- We have this notion of queue being congested or not depending on out of q->nr_requests how many are currently being used. Writeback threads, some filesystems and other places make use of this information to either not to block or to avoid pushing too much of data on device if queue is congested.
With q->nr_requests removed, how do you define queue full and congested semantics?
- I think slee_on_global logic makes sense even without q->nr_requests. Assume that a group allows request descriptor allocation but due to lack of memory, allocation fails. Where do you make this process wait to attempt next time? Making all such failed processes on gloabl list on queue instead of per group list makes more sense to me for following reasons.
- If this is the first request allocation from the group and we make the process sleep on group list, it will never be woken up as no request from that group will complete.
- If there are many processes who failed request descriptor allocation, when some request completes, I think it is more fair to wake these up in FIFO manner to try out allocation again instead of waiting for request to complete from the group process belongs to. The reason being that io controller did not fail the request descriptor allocation.
So even if you get rid of q->nr_requests, you still shall have to have some logic of global wait list where failed allocations can wait.
- It is backward compatible and there are less chances of higher layers being broken due to this.
Gui, I think automatic updation of q->nr_requests is probably not a very good thing. It is user defined tunable and user does not expect this to change automatically.
At this point of time I really can't think of simpler and cleaner way. Ideas are welcome.
Thanks
Vivek
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: [PATCH 21/25] io-controller: Per cgroup request descriptor support
Quoted 37 lines · Gui Jianfeng
Quoted 28 lines · Vivek Goyal
o Currently a request queue has got fixed number of request descriptors for sync and async requests. Once the request descriptors are consumed, new processes are put to sleep and they effectively become serialized. Because sync and async queues are separate, async requests don't impact sync ones but if one is looking for fairness between async requests, that is not achievable if request queue descriptors become bottleneck.
o Make request descriptor's per io group so that if there is lots of IO going on in one cgroup, it does not impact the IO of other group.
o This is just one relatively simple way of doing things. This patch will probably change after the feedback. Folks have raised concerns that in hierchical setup, child's request descriptors should be capped by parent's request descriptors. May be we need to have per cgroup per device files in cgroups where one can specify the upper limit of request descriptors and whenever a cgroup is created one needs to assign request descritor limit making sure total sum of child's request descriptor is not more than of parent.
I guess something like memory controller. Anyway, that would be the next step. For the time being, we have implemented something simpler as follows.
o This patch implements the per cgroup request descriptors. request pool per queue is still common but every group will have its own wait list and its own count of request descriptors allocated to that group for sync and async queues. So effectively
request_listbecomes per io group property and not a global request queue feature.Quoted 7 lines
o Currently one can define
q->nr_requeststo limit request descriptors allocated for the queue. Now there is another tunableq->nr_group_requestswhich controls the requests descriptr limit per group.q->nr_requestssupercedesq->nr_group_requeststo make sure if there are lots of groups present, we don't end up allocating too many request descriptors on the queue.
Quoted 5 lines
Hi Vivek,
In order to prevent
q->nr_requestsfrom becoming the bottle-neck of allocating requests, whether we can updatenr_requestsaccordingly when allocating or removing a cgroup?
Vivek,
I agree with Gui here. In fact, it does not make much sense to keep the nr_requests limit if we already have per cgroup limit in place. This change also simplifies code quite a bit, as we can get rid of all that sleep_on_global logic.
Quoted 6 lines
--
Regards
Gui Jianfeng
PATCH 22/25 io-controller: Per io group bdi congestion interface
In reply to Vivek Goyalo So far there used to be only one pair or queue of request descriptors (one for sync and one for async) per device and number of requests allocated used to decide whether associated bdi is congested or not.
Now with per io group request descriptor infrastructure, there is a pair of request descriptor queue per io group per device. So it might happen that overall request queue is not congested but a particular io group bio belongs to is congested.
Or, it could be otherwise that group is not congested but overall queue is congested. This can happen if user has not properly set the request descriptors limits for queue and groups.
(q->nr_requests < nr_groups * q->nr_group_requests)
Hence there is a need for new interface which can query deivce congestion status per group. This group is determined by the "struct page" IO will be done for. If page is null, then group is determined from the current task context.
o This patch introduces new set of function bdi_*_congested_group(), which take "struct page" as addition argument. These functions will call the block layer and in trun elevator to find out if the io group the page will go into is congested or not.
o Currently I have introduced the core functions and migrated most of the users. But there might be still some left. This is an ongoing TODO item.
o There are some io_get_io_group() related changes which should be pushed into higher patches. Still testing this patch. Will push these changes up in next posting.
--- block/blk-core.c | 21 ++++++++++++++ block/elevator-fq.c | 58 ++++++++++++++++++++++++++++++++++++++++ block/elevator-fq.h | 6 ++++ drivers/md/dm-table.c | 11 +++++--- drivers/md/dm.c | 7 +++-- drivers/md/dm.h | 3 +- drivers/md/linear.c | 7 +++- drivers/md/multipath.c | 7 +++- drivers/md/raid0.c | 6 +++- drivers/md/raid1.c | 9 ++++-- drivers/md/raid10.c | 6 +++- drivers/md/raid5.c | 2 +- fs/afs/write.c | 8 +++++- fs/btrfs/disk-io.c | 6 +++- fs/btrfs/extent_io.c | 12 ++++++++ fs/btrfs/volumes.c | 8 ++++- fs/cifs/file.c | 11 +++++++ fs/ext2/ialloc.c | 2 +- fs/gfs2/aops.c | 12 ++++++++ fs/nilfs2/segbuf.c | 3 +- fs/xfs/linux-2.6/xfs_aops.c | 2 +- fs/xfs/linux-2.6/xfs_buf.c | 2 +- include/linux/backing-dev.h | 61 ++++++++++++++++++++++++++++++++++++++++- include/linux/blkdev.h | 5 +++ mm/backing-dev.c | 62 +++++++++++++++++++++++++++++++++++++++++++ mm/page-writeback.c | 11 +++++++ mm/readahead.c | 2 +- 27 files changed, 318 insertions(+), 32 deletions(-)
block/blk-core.c +21 -0
@@ -90,6 +90,27 @@ void blk_queue_congestion_threshold(struct request_queue *q) q->nr_congestion_off = nr; } +#ifdef CONFIG_GROUP_IOSCHED +int blk_queue_io_group_congested (struct backing_dev_info *bdi,int bdi_bits, +struct page *page) +{ +int ret =0 ; +struct request_queue *q = bdi->unplug_io_data; + +if (!q && !q->elevator) +return bdi_congested (bdi, bdi_bits); + +/* Do we need to hold queue lock? */ +if (bdi_bits & (1 << BDI_sync_congested)) + ret |=elv_io_group_congested (q, page,1 ); + +if (bdi_bits & (1 << BDI_async_congested)) + ret |=elv_io_group_congested (q, page,0 ); + +return ret; +} +#endif +/** * blk_get_backing_dev_info - get the address of a queue's backing_dev_info * @bdev: device
block/elevator-fq.c +58 -0
82 lines not on this page · raw message
block/elevator-fq.h +6 -0
@@ -259,6 +259,10 @@ struct io_group {/* Single ioq per group, used for noop, deadline, anticipatory */ struct io_queue *ioq; +/* io group congestion on and off threshold for request descriptors */ +unsigned int nr_congestion_on; +unsigned int nr_congestion_off; +/* request list associated with the group */ struct request_list rl; };
@@ -531,6 +535,8 @@ extern struct io_queue *elv_lookup_ioq_bio(struct request_queue *q, struct bio *bio);extern struct request_list *io_group_get_request_list (struct request_queue *q,struct bio *bio); +extern int elv_io_group_congested (struct request_queue *q,struct page *page, +int sync);/* Sets the single ioq associated with the io group. (noop, deadline, AS) */ static inline void io_group_set_ioq (struct io_group *iog,struct io_queue *ioq)
drivers/md/dm-table.c +7 -4
@@ -1175,7 +1175,8 @@ int dm_table_resume_targets(struct dm_table *t)return 0 ; } -int dm_table_any_congested (struct dm_table *t,int bdi_bits) +int dm_table_any_congested (struct dm_table *t,int bdi_bits,struct page *page, +int group) {struct dm_dev_internal *dd;struct list_head *devices =dm_table_get_devices (t);
@@ -1185,9 +1186,11 @@ int dm_table_any_congested(struct dm_table *t, int bdi_bits)struct request_queue *q =bdev_get_queue (dd->dm_dev.bdev);char b[BDEVNAME_SIZE ]; -if (likely (q)) - r |=bdi_congested (&q->backing_dev_info, bdi_bits); -else +if (likely (q)) { +struct backing_dev_info *bdi = &q->backing_dev_info; + r |= group ?bdi_congested_group (bdi, bdi_bits, page) + :bdi_congested (bdi, bdi_bits); + }else DMWARN_LIMIT ("%s: any_congested: nonexistent device %s" ,dm_device_name (t->md),bdevname (dd->dm_dev.bdev, b));
drivers/md/dm.c +4 -3
@@ -1608,7 +1608,8 @@ static void dm_unplug_all(struct request_queue *q) } } -static int dm_any_congested (void *congested_data,int bdi_bits) +static int dm_any_congested (void *congested_data,int bdi_bits, +struct page *page,int group) {int r = bdi_bits;struct mapped_device *md = congested_data;
@@ -1625,8 +1626,8 @@ static int dm_any_congested(void *congested_data, int bdi_bits) r = md->queue->backing_dev_info.state & bdi_bits; else - r =dm_table_any_congested (map, bdi_bits); - + r =dm_table_any_congested (map, bdi_bits, page, + group);dm_table_put (map); } }
drivers/md/dm.h +2 -1
@@ -57,7 +57,8 @@ struct list_head *dm_table_get_devices(struct dm_table *t);void dm_table_presuspend_targets (struct dm_table *t);void dm_table_postsuspend_targets (struct dm_table *t);int dm_table_resume_targets (struct dm_table *t); -int dm_table_any_congested (struct dm_table *t,int bdi_bits); +int dm_table_any_congested (struct dm_table *t,int bdi_bits,struct page *page, +int group);int dm_table_any_busy_target (struct dm_table *t);int dm_table_set_type (struct dm_table *t);unsigned dm_table_get_type (struct dm_table *t);
drivers/md/linear.c +5 -2
@@ -102,7 +102,7 @@ static void linear_unplug(struct request_queue *q)rcu_read_unlock (); } -static int linear_congested (void *data,int bits) +static int linear_congested (void *data,int bits,struct page *page,int group) {mddev_t *mddev = data;linear_conf_t *conf;
@@ -113,7 +113,10 @@ static int linear_congested(void *data, int bits)for (i =0 ; i < mddev->raid_disks && !ret ; i++) {struct request_queue *q =bdev_get_queue (conf->disks[i].rdev->bdev); - ret |=bdi_congested (&q->backing_dev_info, bits); +struct backing_dev_info *bdi = &q->backing_dev_info; + + ret |= group ?bdi_congested_group (bdi, bits, page) : +bdi_congested (bdi, bits); }rcu_read_unlock ();
drivers/md/multipath.c +5 -2
@@ -192,7 +192,8 @@ static void multipath_status (struct seq_file *seq, mddev_t *mddev)seq_printf (seq,"]" ); } -static int multipath_congested (void *data,int bits) +static int multipath_congested (void *data,int bits,struct page *page, +int group) {mddev_t *mddev = data;multipath_conf_t *conf = mddev->private;
@@ -203,8 +204,10 @@ static int multipath_congested(void *data, int bits)mdk_rdev_t *rdev =rcu_dereference (conf->multipaths[i].rdev);if (rdev && !test_bit (Faulty, &rdev->flags)) {struct request_queue *q =bdev_get_queue (rdev->bdev); +struct backing_dev_info *bdi = &q->backing_dev_info; - ret |=bdi_congested (&q->backing_dev_info, bits); + ret |= group ?bdi_congested_group (bdi, bits, page) + :bdi_congested (bdi, bits);/* Just like multipath_map, we just check the * first available device */
drivers/md/raid0.c +4 -2
@@ -37,7 +37,7 @@ static void raid0_unplug(struct request_queue *q) } } -static int raid0_congested (void *data,int bits) +static int raid0_congested (void *data,int bits,struct page *page,int group) {mddev_t *mddev = data;raid0_conf_t *conf = mddev->private;
@@ -46,8 +46,10 @@ static int raid0_congested(void *data, int bits)for (i =0 ; i < mddev->raid_disks && !ret ; i++) {struct request_queue *q =bdev_get_queue (devlist[i]->bdev); +struct backing_dev_info *bdi = &q->backing_dev_info; - ret |=bdi_congested (&q->backing_dev_info, bits); + ret |= group ?bdi_congested_group (bdi, bits, page) + :bdi_congested (bdi, bits); }return ret; }
drivers/md/raid1.c +6 -3
@@ -570,7 +570,7 @@ static void raid1_unplug(struct request_queue *q)md_wakeup_thread (mddev->thread); } -static int raid1_congested (void *data,int bits) +static int raid1_congested (void *data,int bits,struct page *page,int group) {mddev_t *mddev = data;conf_t *conf = mddev->private;
@@ -581,14 +581,17 @@ static int raid1_congested(void *data, int bits)mdk_rdev_t *rdev =rcu_dereference (conf->mirrors[i].rdev);if (rdev && !test_bit (Faulty, &rdev->flags)) {struct request_queue *q =bdev_get_queue (rdev->bdev); +struct backing_dev_info *bdi = &q->backing_dev_info;/* Note the '|| 1' - when read_balance prefers * non-congested targets, it can be removed */ if ((bits & (1 <<BDI_async_congested)) ||1 ) - ret |=bdi_congested (&q->backing_dev_info, bits); + ret |= group ?bdi_congested_group (bdi, bits, + page) :bdi_congested (bdi, bits);else - ret &=bdi_congested (&q->backing_dev_info, bits); + ret &= group ?bdi_congested_group (bdi, bits, + page) :bdi_congested (bdi, bits); } }rcu_read_unlock ();
drivers/md/raid10.c +4 -2
@@ -625,7 +625,7 @@ static void raid10_unplug(struct request_queue *q)md_wakeup_thread (mddev->thread); } -static int raid10_congested (void *data,int bits) +static int raid10_congested (void *data,int bits,struct page *page,int group) {mddev_t *mddev = data;conf_t *conf = mddev->private;
@@ -636,8 +636,10 @@ static int raid10_congested(void *data, int bits)mdk_rdev_t *rdev =rcu_dereference (conf->mirrors[i].rdev);if (rdev && !test_bit (Faulty, &rdev->flags)) {struct request_queue *q =bdev_get_queue (rdev->bdev); +struct backing_dev_info *bdi = &q->backing_dev_info; - ret |=bdi_congested (&q->backing_dev_info, bits); + ret |= group ?bdi_congested_group (bdi, bits, page) + :bdi_congested (bdi, bits); } }rcu_read_unlock ();
drivers/md/raid5.c +1 -1
@@ -3323,7 +3323,7 @@ static void raid5_unplug_device(struct request_queue *q)unplug_slaves (mddev); } -static int raid5_congested (void *data,int bits) +static int raid5_congested (void *data,int bits,struct page *page,int group) {mddev_t *mddev = data;raid5_conf_t *conf = mddev->private;
fs/afs/write.c +7 -1
@@ -455,7 +455,7 @@ int afs_writepage(struct page *page, struct writeback_control *wbc) } wbc->nr_to_write -= ret; -if (wbc->nonblocking &&bdi_write_congested (bdi)) +if (wbc->nonblocking &&bdi_or_group_write_congested (bdi, page)) wbc->encountered_congestion =1 ;_leave (" = 0" );
@@ -491,6 +491,12 @@ static int afs_writepages_region(struct address_space *mapping,return 0 ; } +if (wbc->nonblocking &&bdi_write_congested_group (bdi, page)) { + wbc->encountered_congestion =1 ; +page_cache_release (page); +break ; + } +/* at this point we hold neither mapping->tree_lock nor lock on * the page itself: the page may be truncated or invalidated * (changing page->mapping to NULL), or even swizzled back from
fs/btrfs/disk-io.c +4 -2
@@ -1249,7 +1249,8 @@ struct btrfs_root *btrfs_read_fs_root(struct btrfs_fs_info *fs_info,return root; } -static int btrfs_congested_fn (void *congested_data,int bdi_bits) +static int btrfs_congested_fn (void *congested_data,int bdi_bits, +struct page *page,int group) {struct btrfs_fs_info *info = (struct btrfs_fs_info *)congested_data;int ret =0 ;
@@ -1260,7 +1261,8 @@ static int btrfs_congested_fn(void *congested_data, int bdi_bits)if (!device->bdev)continue ; bdi =blk_get_backing_dev_info (device->bdev); -if (bdi &&bdi_congested (bdi, bdi_bits)) { +if (bdi && (group ?bdi_congested_group (bdi, bdi_bits, page) : +bdi_congested (bdi, bdi_bits))) { ret =1 ;break ; }
fs/btrfs/extent_io.c +12 -0
@@ -2368,6 +2368,18 @@ retry:unsigned i; scanned =1 ; + +/* +* If the io group page will go into is congested, bail out. +*/ +if (wbc->nonblocking + &&bdi_write_congested_group (bdi, pvec.pages[0 ])) { + wbc->encountered_congestion =1 ; + done =1 ; +pagevec_release (&pvec); +break ; + } +for (i =0 ; i < nr_pages; i++) {struct page *page = pvec.pages[i];
fs/btrfs/volumes.c +6 -2
@@ -165,6 +165,7 @@ static noinline int run_scheduled_bios(struct btrfs_device *device)unsigned long limit;unsigned long last_waited =0 ;int force_reg =0 ; +struct page *page; bdi =blk_get_backing_dev_info (device->bdev); fs_info = device->dev_root->fs_info;
@@ -276,8 +277,11 @@ loop_lock:* is now congested. Back off and let other work structs * run instead */ -if (pending &&bdi_write_congested (bdi) && batch_run >32 && - fs_info->fs_devices->open_devices >1 ) { +if (pending) + page =bio_iovec_idx (pending,0 )->bv_page; + +if (pending &&bdi_or_group_write_congested (bdi, page) && + num_run >32 && fs_info->fs_devices->open_devices >1 ) {struct io_context *ioc; ioc = current->io_context;
fs/cifs/file.c +11 -0
@@ -1466,6 +1466,17 @@ retry: n_iov =0 ; bytes_to_write =0 ; +/* +* If the io group page will go into is congested, bail out. +*/ +if (wbc->nonblocking && +bdi_write_congested_group (bdi, pvec.pages[0 ])) { + wbc->encountered_congestion =1 ; + done =1 ; +pagevec_release (&pvec); +break ; + } +for (i =0 ; i < nr_pages; i++) { page = pvec.pages[i];/*
fs/ext2/ialloc.c +1 -1
@@ -179,7 +179,7 @@ static void ext2_preread_inode(struct inode *inode)struct backing_dev_info *bdi; bdi = inode->i_mapping->backing_dev_info; -if (bdi_read_congested (bdi)) +if (bdi_or_group_read_congested (bdi,NULL ))return ;if (bdi_write_congested (bdi))return ;
fs/gfs2/aops.c +12 -0
@@ -371,6 +371,18 @@ retry: PAGECACHE_TAG_DIRTY, min(end - index, (pgoff_t)PAGEVEC_SIZE-1) + 1))) { scanned =1 ; + +/* +* If io group page belongs to is congested. bail out. +*/ +if (wbc->nonblocking + &&bdi_write_congested_group (bdi, pvec.pages[0 ])) { + wbc->encountered_congestion =1 ; + done =1 ; +pagevec_release (&pvec); +break ; + } + ret =gfs2_write_jdata_pagevec (mapping, wbc, &pvec, nr_pages, end);if (ret) done =1 ;
fs/nilfs2/segbuf.c +2 -1
@@ -266,8 +266,9 @@ static int nilfs_submit_seg_bio(struct nilfs_write_info *wi, int mode) {struct bio *bio = wi->bio;int err; +struct page *page =bio_iovec_idx (bio,0 )->bv_page; -if (wi->nbio >0 &&bdi_write_congested (wi->bdi)) { +if (wi->nbio >0 &&bdi_or_group_write_congested (wi->bdi, page)) {wait_for_completion (&wi->bio_event); wi->nbio--;if (unlikely (atomic_read (&wi->err))) {
fs/xfs/linux-2.6/xfs_aops.c +1 -1
@@ -891,7 +891,7 @@ xfs_convert_page( bdi = inode->i_mapping->backing_dev_info; wbc->nr_to_write--; -if (bdi_write_congested (bdi)) { +if (bdi_or_group_write_congested (bdi, page)) { wbc->encountered_congestion =1 ; done =1 ; }else if (wbc->nr_to_write <=0 ) {
fs/xfs/linux-2.6/xfs_buf.c +1 -1
@@ -714,7 +714,7 @@ xfs_buf_readahead(struct backing_dev_info *bdi; bdi = target->bt_mapping->backing_dev_info; -if (bdi_read_congested (bdi)) +if (bdi_or_group_read_congested (bdi,NULL ))return ; flags |= (XBF_TRYLOCK |XBF_ASYNC |XBF_READ_AHEAD );
include/linux/backing-dev.h +59 -2
79 lines not on this page · raw message
include/linux/blkdev.h +5 -0
@@ -897,6 +897,11 @@ static inline void blk_set_queue_congested(struct request_queue *q, int rw)set_bdi_congested (&q->backing_dev_info, rw); } +#ifdef CONFIG_GROUP_IOSCHED +extern int blk_queue_io_group_congested (struct backing_dev_info *bdi, +int bdi_bits,struct page *page); +#endif +extern void blk_start_queue (struct request_queue *q);extern void blk_stop_queue (struct request_queue *q);extern void blk_sync_queue (struct request_queue *q);
mm/backing-dev.c +62 -0
71 lines not on this page · raw message
mm/page-writeback.c +11 -0
@@ -982,6 +982,17 @@ retry:if (nr_pages ==0 )break ; +/* +* If the io group page will go into is congested, bail out. +*/ +if (wbc->nonblocking + &&bdi_write_congested_group (bdi, pvec.pages[0 ])) { + wbc->encountered_congestion =1 ; + done =1 ; +pagevec_release (&pvec); +break ; + } +for (i =0 ; i < nr_pages; i++) {struct page *page = pvec.pages[i];
mm/readahead.c +1 -1
@@ -542,7 +542,7 @@ page_cache_async_readahead(struct address_space *mapping,/* * Defer asynchronous read-ahead on IO congestion. */ -if (bdi_read_congested (mapping->backing_dev_info)) +if (bdi_or_group_read_congested (mapping->backing_dev_info,NULL ))return ;/* do read-ahead */
Signature · 1 line
1.6.0.6
Re: [PATCH 22/25] io-controller: Per io group bdi congestion interface
Hi,
Vivek Goyal wrote, on 07/02/2009 04:01 PM:
Quoted 7 lines
block/blk-core.c +0
-0@@ -90,6 +90,27 @@ void blk_queue_congestion_threshold(struct request_queue *q) q->nr_congestion_off = nr; }
block/blk-core.c void blk_queue_congestion_threshold(struct request_queue *q) · line 101
Quoted 10 lines
block/blk-core.c +9
-0+#ifdef CONFIG_GROUP_IOSCHED +int blk_queue_io_group_congested (struct backing_dev_info *bdi,int bdi_bits, +struct page *page) +{ +int ret =0 ; +struct request_queue *q = bdi->unplug_io_data; + +if (!q&& !q->elevator) +return bdi_congested (bdi, bdi_bits);
It causes NULL pointer dereference for brd etc.
--- block/blk-core.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)
block/blk-core.c +1 -1
@@ -97,7 +97,7 @@ int blk_queue_io_group_congested(struct backing_dev_info *bdi, int bdi_bits,int ret =0 ;struct request_queue *q = bdi->unplug_io_data; -if (!q && !q->elevator) +if (!q || !q->elevator)return bdi_congested (bdi, bdi_bits);/* Do we need to hold queue lock? */
Signature · 7 lines
1.6.2.5
--
IKEDA, Munehiro
NEC Corporation of America
[email protected]Re: [PATCH 22/25] io-controller: Per io group bdi congestion interface
Quoted 33 lines · Munehiro Ikeda
Hi,
Vivek Goyal wrote, on 07/02/2009 04:01 PM:
Quoted 7 lines
block/blk-core.c +0
-0@@ -90,6 +90,27 @@ void blk_queue_congestion_threshold(struct request_queue *q) q->nr_congestion_off = nr; }block/blk-core.c void blk_queue_congestion_threshold(struct request_queue *q) · line 101
Quoted 10 lines
block/blk-core.c +9
-0+#ifdef CONFIG_GROUP_IOSCHED +int blk_queue_io_group_congested (struct backing_dev_info *bdi,int bdi_bits, +struct page *page) +{ +int ret =0 ; +struct request_queue *q = bdi->unplug_io_data; + +if (!q&& !q->elevator) +return bdi_congested (bdi, bdi_bits);It causes NULL pointer dereference for brd etc.
Signed-off-by Munehiro "Muuhh" Ikeda <[email protected]>--- block/blk-core.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)block/blk-core.c +0
-0
block/blk-core.c int blk_queue_io_group_congested(struct backing_dev_info *bdi, int bdi_bits, · line 102
Quoted 7 lines
block/blk-core.c +1
-1@@ -97,7 +97,7 @@ int blk_queue_io_group_congested(struct backing_dev_info *bdi, int bdi_bits,int ret =0 ;struct request_queue *q = bdi->unplug_io_data; -if (!q && !q->elevator) +if (!q || !q->elevator)return bdi_congested (bdi, bdi_bits);
Hi,
Thanks for the patch. I also noticed this recently and fixed it for next to be posted version.
Thanks
Vivek
Quoted 9 lines
block/blk-core.c +0
-0/* Do we need to hold queue lock? */ Signature · 7 lines
1.6.2.5 -- IKEDA, Munehiro NEC Corporation of America [email protected]
A message that is not in the archive (sent off-list or not yet received): 4A5FC2CA.1040609-MDRzhb/z0dd8UrSeD/[email protected]
Re: [PATCH 22/25] io-controller: Per io group bdi congestion interface
Quoted 33 lines · Munehiro Ikeda
Hi,
Vivek Goyal wrote, on 07/02/2009 04:01 PM:
Quoted 7 lines
block/blk-core.c +0
-0@@ -90,6 +90,27 @@ void blk_queue_congestion_threshold(struct request_queue *q) q->nr_congestion_off = nr; }block/blk-core.c void blk_queue_congestion_threshold(struct request_queue *q) · line 101
Quoted 10 lines
block/blk-core.c +9
-0+#ifdef CONFIG_GROUP_IOSCHED +int blk_queue_io_group_congested (struct backing_dev_info *bdi,int bdi_bits, +struct page *page) +{ +int ret =0 ; +struct request_queue *q = bdi->unplug_io_data; + +if (!q&& !q->elevator) +return bdi_congested (bdi, bdi_bits);It causes NULL pointer dereference for brd etc.
--- block/blk-core.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)block/blk-core.c +0
-0
block/blk-core.c int blk_queue_io_group_congested(struct backing_dev_info *bdi, int bdi_bits, · line 102
Quoted 7 lines
block/blk-core.c +1
-1@@ -97,7 +97,7 @@ int blk_queue_io_group_congested(struct backing_dev_info *bdi, int bdi_bits,int ret =0 ;struct request_queue *q = bdi->unplug_io_data; -if (!q && !q->elevator) +if (!q || !q->elevator)return bdi_congested (bdi, bdi_bits);
Hi,
Thanks for the patch. I also noticed this recently and fixed it for next to be posted version.
Thanks
Vivek
Quoted 9 lines
block/blk-core.c +0
-0/* Do we need to hold queue lock? */ Signature · 7 lines
1.6.2.5 -- IKEDA, Munehiro NEC Corporation of America m-ikeda-MDRzhb/z0dd8UrSeD/[email protected]
A message that is not in the archive (sent off-list or not yet received): 1246564917-19603-23-git-send-email-vgoyal-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org
Re: [PATCH 22/25] io-controller: Per io group bdi congestion interface
Hi,
Vivek Goyal wrote, on 07/02/2009 04:01 PM:
Quoted 7 lines
block/blk-core.c +0
-0@@ -90,6 +90,27 @@ void blk_queue_congestion_threshold(struct request_queue *q) q->nr_congestion_off = nr; }
block/blk-core.c void blk_queue_congestion_threshold(struct request_queue *q) · line 101
Quoted 10 lines
block/blk-core.c +9
-0+#ifdef CONFIG_GROUP_IOSCHED +int blk_queue_io_group_congested (struct backing_dev_info *bdi,int bdi_bits, +struct page *page) +{ +int ret =0 ; +struct request_queue *q = bdi->unplug_io_data; + +if (!q&& !q->elevator) +return bdi_congested (bdi, bdi_bits);
It causes NULL pointer dereference for brd etc.
--- block/blk-core.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)
block/blk-core.c +1 -1
@@ -97,7 +97,7 @@ int blk_queue_io_group_congested(struct backing_dev_info *bdi, int bdi_bits,int ret =0 ;struct request_queue *q = bdi->unplug_io_data; -if (!q && !q->elevator) +if (!q || !q->elevator)return bdi_congested (bdi, bdi_bits);/* Do we need to hold queue lock? */
Signature · 7 lines
1.6.2.5
--
IKEDA, Munehiro
NEC Corporation of America
m-ikeda-MDRzhb/z0dd8UrSeD/[email protected]PATCH 23/25 io-controller: Support per cgroup per device weights and io class
In reply to Vivek GoyalThis patch enables per-cgroup per-device weight and ioprio_class handling. A new cgroup interface "policy" is introduced. You can make use of this file to configure weight and ioprio_class for each device in a given cgroup. The original "weight" and "ioprio_class" files are still available. If you don't do special configuration for a particular device, "weight" and "ioprio_class" are used as default values in this device.
You can use the following format to play with the new interface. #echo dev_major:dev_minor weight ioprio_class > /patch/to/cgroup/policy weight=0 means removing the policy for device.
Examples:
Configure weight=300 ioprio_class=2 on /dev/hdb (8:16) in this cgroup
# echo "8:16 300 2" > io.policy # cat io.policy dev weight class 8:16 300 2
Configure weight=500 ioprio_class=1 on /dev/hda (8:0) in this cgroup
# echo "8:0 500 1" > io.policy # cat io.policy dev weight class 8:0 500 1 8:16 300 2
Remove the policy for /dev/hda in this cgroup
# echo 8:0 0 1 > io.policy # cat io.policy dev weight class 8:16 300 2
Changelog (v1 -> v2)
- Rename some structures
- Use spin_lock_irqsave() and spin_lock_irqrestore() version to prevent from enabling the interrupts unconditionally.
- Fix policy setup bug when switching to another io scheduler.
- If a policy is available for a specific device, don't update weight and io class when writing "weight" and "iprio_class".
- Fix a bug when parsing policy string.
--- block/elevator-fq.c | 266 ++++++++++++++++++++++++++++++++++++++++++++++++++- block/elevator-fq.h | 10 ++ 2 files changed, 272 insertions(+), 4 deletions(-)
block/elevator-fq.c +262 -4
@@ -17,6 +17,7 @@#include <linux/blktrace_api.h> #include <linux/seq_file.h> #include <linux/biotrack.h> +#include <linux/genhd.h> /* Values taken from cfq */ const int elv_slice_sync =HZ /10 ;
@@ -1053,12 +1054,31 @@ static void bfq_init_entity(struct io_entity *entity, struct io_group *iog) entity->sched_data = &iog->sched_data; } -static void io_group_init_entity (struct io_cgroup *iocg,struct io_group *iog) +static struct io_policy_node *policy_search_node (const struct io_cgroup *iocg, +dev_t dev); + +static void io_group_init_entity (struct io_cgroup *iocg,struct io_group *iog, +dev_t dev) {struct io_entity *entity = &iog->entity; +struct io_policy_node *pn; +unsigned long flags; + +spin_lock_irqsave (&iocg->lock, flags); + pn =policy_search_node (iocg, dev); +if (pn) { + entity->weight = pn->weight; + entity->new_weight = pn->weight; + entity->ioprio_class = pn->ioprio_class; + entity->new_ioprio_class = pn->ioprio_class; + }else { + entity->weight = iocg->weight; + entity->new_weight = iocg->weight; + entity->ioprio_class = iocg->ioprio_class; + entity->new_ioprio_class = iocg->ioprio_class; + } +spin_unlock_irqrestore (&iocg->lock, flags); - entity->weight = entity->new_weight = iocg->weight; - entity->ioprio_class = entity->new_ioprio_class = iocg->ioprio_class; entity->ioprio_changed =1 ; entity->my_sched_data = &iog->sched_data; }
@@ -1174,6 +1194,227 @@ io_cgroup_lookup_group(struct io_cgroup *iocg, void *key)return NULL ; } +static int io_cgroup_policy_read (struct cgroup *cgrp,struct cftype *cft, +struct seq_file *m) +{ +struct io_cgroup *iocg; +struct io_policy_node *pn; + + iocg =cgroup_to_io_cgroup (cgrp); + +if (list_empty (&iocg->policy_list)) + goto out; + +seq_printf (m,"dev\tweight\tclass\n" ); + +spin_lock_irq (&iocg->lock); +list_for_each_entry (pn, &iocg->policy_list, node) { +seq_printf (m,"%u:%u\t%u\t%hu\n" ,MAJOR (pn->dev), +MINOR (pn->dev), pn->weight, pn->ioprio_class); + } +spin_unlock_irq (&iocg->lock); +out: +return 0 ; +} + +static inline void policy_insert_node (struct io_cgroup *iocg, +struct io_policy_node *pn) +{ +list_add (&pn->node, &iocg->policy_list); +} + +/* Must be called with iocg->lock held */ +static inline void policy_delete_node (struct io_policy_node *pn) +{ +list_del (&pn->node); +} + +/* Must be called with iocg->lock held */ +static struct io_policy_node *policy_search_node (const struct io_cgroup *iocg, +dev_t dev) +{ +struct io_policy_node *pn; + +if (list_empty (&iocg->policy_list)) +return NULL ; + +list_for_each_entry (pn, &iocg->policy_list, node) { +if (pn->dev == dev) +return pn; + } + +return NULL ; +} + +static int check_dev_num (dev_t dev) +{ +int part =0 ; +struct gendisk *disk; + + disk =get_gendisk (dev, &part); +if (!disk || part) +return -ENODEV ; + +return 0 ; +} + +static int policy_parse_and_set (char *buf,struct io_policy_node *newpn) +{ +char *s[4 ], *p, *major_s =NULL , *minor_s =NULL ; +int ret; +unsigned long major, minor, temp; +int i =0 ; +dev_t dev; + +memset (s,0 ,sizeof (s)); +while ((p =strsep (&buf," " )) !=NULL ) { +if (!*p) +continue ; + s[i++] = p; + +/* Prevent from inputing too many things */ +if (i ==4 ) +break ; + } + +if (i !=3 ) +return -EINVAL ; + + p =strsep (&s[0 ],":" ); +if (p !=NULL ) + major_s = p; +else +return -EINVAL ; + + minor_s = s[0 ]; +if (!minor_s) +return -EINVAL ; + + ret =strict_strtoul (major_s,10 , &major); +if (ret) +return -EINVAL ; + + ret =strict_strtoul (minor_s,10 , &minor); +if (ret) +return -EINVAL ; + + dev =MKDEV (major, minor); + + ret =check_dev_num (dev); +if (ret) +return ret; + + newpn->dev = dev; + +if (s[1 ] ==NULL ) +return -EINVAL ; + + ret =strict_strtoul (s[1 ],10 , &temp); +if (ret || temp >WEIGHT_MAX ) +return -EINVAL ; + + newpn->weight = temp; + +if (s[2 ] ==NULL ) +return -EINVAL ; + + ret =strict_strtoul (s[2 ],10 , &temp); +if (ret || temp <IOPRIO_CLASS_RT || temp >IOPRIO_CLASS_IDLE ) +return -EINVAL ; + newpn->ioprio_class = temp; + +return 0 ; +} + +static int io_cgroup_policy_write (struct cgroup *cgrp,struct cftype *cft, +const char *buffer) +{ +struct io_cgroup *iocg; +struct io_policy_node *newpn, *pn; +char *buf; +int ret =0 ; +int keep_newpn =0 ; +struct hlist_node *n; +struct io_group *iog; + + buf =kstrdup (buffer,GFP_KERNEL ); +if (!buf) +return -ENOMEM ; + + newpn =kzalloc (sizeof (*newpn),GFP_KERNEL ); +if (!newpn) { + ret = -ENOMEM ; + goto free_buf; + } + + ret =policy_parse_and_set (buf, newpn); +if (ret) + goto free_newpn; + +if (!cgroup_lock_live_group (cgrp)) { + ret = -ENODEV ; + goto free_newpn; + } + + iocg =cgroup_to_io_cgroup (cgrp); +spin_lock_irq (&iocg->lock); + + pn =policy_search_node (iocg, newpn->dev); +if (!pn) { +if (newpn->weight !=0 ) { +policy_insert_node (iocg, newpn); + keep_newpn =1 ; + } + goto update_io_group; + } + +if (newpn->weight ==0 ) { +/* weight == 0 means deleteing a policy */ +policy_delete_node (pn); + goto update_io_group; + } + + pn->weight = newpn->weight; + pn->ioprio_class = newpn->ioprio_class; + +update_io_group: +hlist_for_each_entry (iog, n, &iocg->group_data, group_node) { +if (iog->dev == newpn->dev) { +if (newpn->weight) { + iog->entity.new_weight = newpn->weight; + iog->entity.new_ioprio_class = + newpn->ioprio_class; +/* +* iog weight and ioprio_class updating +* actually happens if ioprio_changed is set. +* So ensure ioprio_changed is not set until +* new weight and new ioprio_class are updated. +*/ +smp_wmb (); + iog->entity.ioprio_changed =1 ; + }else { + iog->entity.new_weight = iocg->weight; + iog->entity.new_ioprio_class = + iocg->ioprio_class; + +/* The same as above */ +smp_wmb (); + iog->entity.ioprio_changed =1 ; + } + } + } +spin_unlock_irq (&iocg->lock); + +cgroup_unlock (); + +free_newpn: +if (!keep_newpn) +kfree (newpn); +free_buf: +kfree (buf); +return ret; +} +#define SHOW_FUNCTION (__VAR) \ static u64 io_cgroup_##__VAR##_read(struct cgroup *cgroup, \ struct cftype *cftype) \
@@ -1206,6 +1447,7 @@ static int io_cgroup_##__VAR##_write(struct cgroup *cgroup, \struct io_cgroup *iocg; \struct io_group *iog; \struct hlist_node *n; \ +struct io_policy_node *pn; \ \if (val < (__MIN) || val > (__MAX)) \return -EINVAL ; \
@@ -1218,6 +1460,9 @@ static int io_cgroup_##__VAR##_write(struct cgroup *cgroup, \spin_lock_irq (&iocg->lock); \ iocg->__VAR = (unsigned long )val; \ hlist_for_each_entry(iog, n, &iocg->group_data, group_node) { \ + pn =policy_search_node (iocg, iog->dev); \ +if (pn) \ +continue ; \ iog->entity.new_##__VAR = (unsigned long)val; \smp_wmb (); \ iog->entity.ioprio_changed =1 ; \
@@ -1295,6 +1540,12 @@ static int io_cgroup_disk_sectors_read(struct cgroup *cgroup,struct cftype bfqio_files[] = { { + .name ="policy" , + .read_seq_string = io_cgroup_policy_read, + .write_string = io_cgroup_policy_write, + .max_write_len =256 , + }, + { .name ="weight" , .read_u64 = io_cgroup_weight_read, .write_u64 = io_cgroup_weight_write,
@@ -1336,6 +1587,7 @@ static struct cgroup_subsys_state *iocg_create(struct cgroup_subsys *subsys,INIT_HLIST_HEAD (&iocg->group_data); iocg->weight =IO_DEFAULT_GRP_WEIGHT ; iocg->ioprio_class =IO_DEFAULT_GRP_CLASS ; +INIT_LIST_HEAD (&iocg->policy_list);return &iocg->css; }
@@ -1438,7 +1690,7 @@ io_group_chain_alloc(struct request_queue *q, void *key, struct cgroup *cgroup)sscanf (dev_name (bdi->dev),"%u:%u" , &major, &minor); iog->dev =MKDEV (major, minor); -io_group_init_entity (iocg, iog); +io_group_init_entity (iocg, iog, iog->dev); iog->my_entity = &iog->entity;atomic_set (&iog->ref,0 );
@@ -1904,6 +2156,7 @@ static void iocg_destroy(struct cgroup_subsys *subsys, struct cgroup *cgroup)struct io_group *iog;struct elv_fq_data *efqd;unsigned long uninitialized_var (flags ); +struct io_policy_node *pn, *pntmp;/* * io groups are linked in two lists. One list is maintained
@@ -1943,6 +2196,11 @@ remove_entry: goto remove_entry; done: +list_for_each_entry_safe (pn, pntmp, &iocg->policy_list, node) { +policy_delete_node (pn); +kfree (pn); + } +free_css_id (&io_subsys, &iocg->css);rcu_read_unlock ();BUG_ON (!hlist_empty (&iocg->group_data));
block/elevator-fq.h +10 -0
@@ -267,6 +267,13 @@ struct io_group {struct request_list rl; }; +struct io_policy_node { +struct list_head node; +dev_t dev; +unsigned int weight; +unsigned short ioprio_class; +}; +/** * struct io_cgroup - io cgroup data structure. * @css: subsystem state for io in the containing cgroup.
@@ -284,6 +291,9 @@ struct io_cgroup {unsigned int weight;unsigned short ioprio_class; +/* list of io_policy_node */ +struct list_head policy_list; +spinlock_t lock;struct hlist_head group_data; };
Signature · 1 line
1.6.0.6
PATCH 24/25 io-controller: Debug hierarchical IO scheduling
In reply to Vivek Goyalo Littile debugging aid for hierarchical IO scheduling.
o Enabled under CONFIG_DEBUG_GROUP_IOSCHED
o Currently it outputs more debug messages in blktrace output which helps a great deal in debugging in hierarchical setup. It also creates additional cgroup interfaces io.disk_queue and io.disk_dequeue to output some more debugging data.
--- block/Kconfig.iosched | 10 ++- block/as-iosched.c | 50 ++++++--- block/elevator-fq.c | 280 ++++++++++++++++++++++++++++++++++++++++++++++++- block/elevator-fq.h | 36 +++++++ 4 files changed, 354 insertions(+), 22 deletions(-)
block/Kconfig.iosched +9 -1
@@ -140,6 +140,14 @@ config TRACK_ASYNC_CONTEXT request, original owner of the bio is decided by using io tracking patches otherwise we continue to attribute the request to the submitting thread. -endmenu +config DEBUG_GROUP_IOSCHED + bool "Debug Hierarchical Scheduling support" + depends on CGROUPS && GROUP_IOSCHED + default n + ---help--- + Enable some debugging hooks for hierarchical scheduling support. + Currently it just outputs more information in blktrace output. + +endmenu endif
block/as-iosched.c +32 -18
152 lines not on this page · raw message
block/elevator-fq.c +277 -3
385 lines not on this page · raw message
block/elevator-fq.h +36 -0
@@ -265,6 +265,23 @@ struct io_group {/* request list associated with the group */ struct request_list rl; + +#ifdef CONFIG_DEBUG_GROUP_IOSCHED +/* How many times this group has been added to active tree */ +unsigned long queue; + +/* How long this group remained on active tree, in ms */ +unsigned long queue_duration; + +/* When was this group added to active tree */ +unsigned long queue_start; + +/* How many times this group has been removed from active tree */ +unsigned long dequeue; + +/* Store cgroup path */ +char path[128 ]; +#endif };struct io_policy_node {
@@ -368,10 +385,29 @@ struct elv_fq_data { };/* Logging facilities. */ +#ifdef CONFIG_DEBUG_GROUP_IOSCHED +#define elv_log_ioq(efqd, ioq, fmt, args...) \ +{ \ + blk_add_trace_msg((efqd)->queue, "elv%d%c %s " fmt, (ioq)->pid, \ + elv_ioq_sync(ioq) ? 'S' : 'A', \ + ioq_to_io_group(ioq)->path, ##args); \ +} + +#define elv_log_iog(efqd, iog, fmt, args...) \ +{ \ + blk_add_trace_msg((efqd)->queue, "elv %s " fmt, (iog)->path, ##args); \ +} + +#else #define elv_log_ioq(efqd, ioq, fmt, args...) \ blk_add_trace_msg((efqd)->queue, "elv%d%c " fmt, (ioq)->pid, \ elv_ioq_sync(ioq) ? 'S' : 'A', ##args) +#define elv_log_iog(efqd, iog, fmt, args...) \ + blk_add_trace_msg((efqd)->queue, "elv " fmt, ##args) + +#endif + #define elv_log(efqd, fmt, args...) \ blk_add_trace_msg((efqd)->queue, "elv " fmt, ##args)
Signature · 1 line
1.6.0.6
PATCH 25/25 io-controller: experimental debug patch for async queue wait before expiry
In reply to Vivek Goyalo A debug patch which does wait for next IO from async queue once it becomes empty.
o For async writes, traffic seen by IO scheduler is not in proportion to the weight of the cgroup task/page belongs to. So if there are two processes doing heavy writeouts in two cgroups with weights 1000 and 500 respectively, then IO scheduler does not see more traffic/IO from higher weight cgroup even if IO scheduler tries to give it higher disk time. Effectively, the async queue belonging to higher weight cgroup becomes empty, and gets out of contention for disk and lower weight cgroup gets to use disk giving an impression in user space that higher weight cgroup did not get higher time to disk.
o This is more of a problem at page cache level where a higher weight process might be writing out the pages of lower weight process etc and should be fixed there.
o While we fix those issues, introducing this debug patch which allows one
to idle on async queue (tunable via /sys/blolc/<disk>/queue/async_slice_idle) so that once a higher weight queue becomes empty, instead of expiring it we try to wait for next request to come from that queue hence giving it
higher disk time. A higher value of async_slice_idle, around 300ms, helps
me get some right numbers for my setup. Note: higher disk time would not
necessarily translate in more IO done as higher weight group is not pushing
enough IO to io scheduler. It is just a debugging aid to prove correctness
of IO controller by providing higher disk times to higher weight cgroup.
--- block/cfq-iosched.c | 1 + block/elevator-fq.c | 39 ++++++++++++++++++++++++++++++++++++--- block/elevator-fq.h | 5 +++++ 3 files changed, 42 insertions(+), 3 deletions(-)
block/cfq-iosched.c +1 -0
@@ -2093,6 +2093,7 @@ static struct elv_fs_entry cfq_attrs[] = {ELV_ATTR (slice_sync),ELV_ATTR (slice_async),ELV_ATTR (fairness), +ELV_ATTR (async_slice_idle), __ATTR_NULL };
block/elevator-fq.c +36 -3
@@ -24,6 +24,7 @@ const int elv_slice_sync = HZ / 10;int elv_slice_async =HZ /25 ;const int elv_slice_async_rq =2 ;int elv_slice_idle =HZ /125 ; +int elv_async_slice_idle =0 ;static struct kmem_cache *elv_ioq_pool;/* Maximum Window length for updating average disk rate */
@@ -2819,6 +2820,8 @@ SHOW_FUNCTION(elv_slice_async_show, efqd->elv_slice[0], 1);EXPORT_SYMBOL (elv_slice_async_show);SHOW_FUNCTION (elv_fairness_show, efqd->fairness,0 );EXPORT_SYMBOL (elv_fairness_show); +SHOW_FUNCTION (elv_async_slice_idle_show, efqd->elv_async_slice_idle,1 ); +EXPORT_SYMBOL (elv_async_slice_idle_show);#undef SHOW_FUNCTION#define STORE_FUNCTION (__FUNC, __PTR,MIN ,MAX , __CONV) \
@@ -2845,6 +2848,8 @@ STORE_FUNCTION(elv_slice_async_store, &efqd->elv_slice[0], 1, UINT_MAX, 1);EXPORT_SYMBOL (elv_slice_async_store);STORE_FUNCTION (elv_fairness_store, &efqd->fairness,0 ,2 ,0 );EXPORT_SYMBOL (elv_fairness_store); +STORE_FUNCTION (elv_async_slice_idle_store, &efqd->elv_async_slice_idle,0 ,UINT_MAX ,1 ); +EXPORT_SYMBOL (elv_async_slice_idle_store);#undef STORE_FUNCTIONvoid elv_schedule_dispatch (struct request_queue *q)
@@ -3018,7 +3023,7 @@ int elv_init_ioq(struct elevator_queue *eq, struct io_queue *ioq, ioq->pid = current->pid; ioq->sched_queue = sched_queue; -if (is_sync && !elv_ioq_class_idle (ioq)) +if (!elv_ioq_class_idle (ioq) && (is_sync || efqd->fairness))elv_mark_ioq_idle_window (ioq);bfq_init_entity (&ioq->entity, iog); ioq->entity.budget =elv_prio_to_slice (efqd, ioq);
@@ -3699,7 +3704,12 @@ static void elv_ioq_arm_slice_timer(struct request_queue *q, int wait_for_busy)/* * idle is disabled, either manually or by past process history */ -if (!efqd->elv_slice_idle || !elv_ioq_idle_window (ioq)) +if ((elv_ioq_sync (ioq) && !efqd->elv_slice_idle) || + !elv_ioq_idle_window (ioq)) +return ; + +/* If this is async queue and async_slice_idle is disabled, return */ +if (!elv_ioq_sync (ioq) && !efqd->elv_async_slice_idle)return ;/*
@@ -3708,7 +3718,10 @@ static void elv_ioq_arm_slice_timer(struct request_queue *q, int wait_for_busy)*/ if (wait_for_busy) {elv_mark_ioq_wait_busy (ioq); - sl = efqd->elv_slice_idle; +if (elv_ioq_sync (ioq)) + sl = efqd->elv_slice_idle; +else + sl = efqd->elv_async_slice_idle;mod_timer (&efqd->idle_slice_timer, jiffies + sl);elv_log_ioq (efqd, ioq,"arm idle: %lu wait busy=1" , sl);return ;
@@ -3882,6 +3895,18 @@ void *elv_fq_select_ioq(struct request_queue *q, int force) goto keep_queue; } +/* +* If this is an async queue which has time slice left but not +* requests. Wait busy is also not on (may be because when last +* request completed, ioq was not empty). Wait for the request +* completion. May be completion will turn wait busy on. +*/ +if (efqd->fairness && efqd->elv_async_slice_idle && !elv_ioq_sync (ioq) + &&elv_ioq_nr_dispatched (ioq)) { + ioq =NULL ; + goto keep_queue; + } + slice_expired =0 ; expire: if (efqd->fairness >= 2 && !force && ioq && ioq->dispatched
@@ -4076,6 +4101,13 @@ void elv_ioq_completed_request(struct request_queue *q, struct request *rq) goto done; } +/* For async queue try to do wait busy */ +if (efqd->fairness && !elv_ioq_sync (ioq) && !ioq->nr_queued + && (elv_iog_nr_active (iog) <=1 )) { +elv_ioq_arm_slice_timer (q,1 ); + goto done; + } +/* * If there are no requests waiting in this queue, and * there are other queues ready to issue requests, AND
@@ -4215,6 +4247,7 @@ int elv_init_fq_data(struct request_queue *q, struct elevator_queue *e) efqd->elv_slice[0 ] = elv_slice_async; efqd->elv_slice[1 ] = elv_slice_sync; efqd->elv_slice_idle = elv_slice_idle; + efqd->elv_async_slice_idle = elv_async_slice_idle; efqd->hw_tag =1 ;/* For the time being keep fairness enabled by default */
block/elevator-fq.h +5 -0
@@ -362,6 +362,8 @@ struct elv_fq_data {* users of this functionality. */ unsigned int elv_slice_idle; +/* idle slice for async queue */ +unsigned int elv_async_slice_idle;struct timer_list idle_slice_timer;struct work_struct unplug_work;
@@ -647,6 +649,9 @@ extern ssize_t elv_slice_async_store(struct elevator_queue *q, const char *name,extern ssize_t elv_fairness_show (struct elevator_queue *q,char *name);extern ssize_t elv_fairness_store (struct elevator_queue *q,const char *name,size_t count); +extern ssize_t elv_async_slice_idle_show (struct elevator_queue *q,char *name); +extern ssize_t elv_async_slice_idle_store (struct elevator_queue *q, +const char *name,size_t count);/* Functions used by elevator.c */ extern int elv_init_fq_data (struct request_queue *q,struct elevator_queue *e);
Signature · 1 line
1.6.0.6
Quoted 7 lines · Vivek Goyal
Hi All,
Here is the V6 of the IO controller patches generated on top of 2.6.31-rc1.
Previous versions of the patches was posted here.
Quoted 9 lines
(V1) http://lkml.org/lkml/2009/3/11/486
(V2) http://lkml.org/lkml/2009/5/5/275
(V3) http://lkml.org/lkml/2009/5/26/472
(V4) http://lkml.org/lkml/2009/6/8/580
(V5) http://lkml.org/lkml/2009/6/19/279This patchset is still work in progress but I want to keep on getting the snapshot of my tree out at regular intervals to get the feedback hence V6.
Hi, Vivek,
I was able to compile and boot a 2.6.31-rc1 kernel with this patchset. I have a request could you fold up all patches and make one consolidated patch available somewhere (makes it easier to test), may be a git tree?
I did some quick tests with some io benchmarks and found in a simple scenario that the scheduler worked as expected, except that it took very long. I'll investigate further and revert back.
Signature · 1 line
Balbir
Quoted 19 lines · Balbir Singh
Quoted 7 lines · Vivek Goyal
Hi All,
Here is the V6 of the IO controller patches generated on top of 2.6.31-rc1.
Previous versions of the patches was posted here.
Quoted 9 lines
(V1) http://lkml.org/lkml/2009/3/11/486
(V2) http://lkml.org/lkml/2009/5/5/275
(V3) http://lkml.org/lkml/2009/5/26/472
(V4) http://lkml.org/lkml/2009/6/8/580
(V5) http://lkml.org/lkml/2009/6/19/279This patchset is still work in progress but I want to keep on getting the snapshot of my tree out at regular intervals to get the feedback hence V6.
Quoted 7 lines
Hi, Vivek,
I was able to compile and boot a 2.6.31-rc1 kernel with this patchset. I have a request could you fold up all patches and make one consolidated patch available somewhere (makes it easier to test), may be a git tree?
Thanks for trying it out balbir. Ok, for ease of patching and testing, I will also maintain a consolidated patch. For V6 you can download the patch from here.
http://people.redhat.com/~vgoyal/io-controller/io-scheduler-based-io-controller-v6.patch
Quoted 3 lines
I did some quick tests with some io benchmarks and found in a simple scenario that the scheduler worked as expected, except that it took very long. I'll investigate further and revert back.
Thanks. I will wait for details.
Vivek
Quoted 28 lines · Vivek Goyal
Quoted 19 lines · Balbir Singh
Quoted 7 lines · Vivek Goyal
Hi All,
Here is the V6 of the IO controller patches generated on top of 2.6.31-rc1.
Previous versions of the patches was posted here.
Quoted 9 lines
(V1) http://lkml.org/lkml/2009/3/11/486
(V2) http://lkml.org/lkml/2009/5/5/275
(V3) http://lkml.org/lkml/2009/5/26/472
(V4) http://lkml.org/lkml/2009/6/8/580
(V5) http://lkml.org/lkml/2009/6/19/279This patchset is still work in progress but I want to keep on getting the snapshot of my tree out at regular intervals to get the feedback hence V6.
Quoted 7 lines
Hi, Vivek,
I was able to compile and boot a 2.6.31-rc1 kernel with this patchset. I have a request could you fold up all patches and make one consolidated patch available somewhere (makes it easier to test), may be a git tree?
Quoted 6 lines
Thanks for trying it out balbir. Ok, for ease of patching and testing, I will also maintain a consolidated patch. For V6 you can download the patch from here.
http://people.redhat.com/~vgoyal/io-controller/io-scheduler-based-io-controller-v6.patch
Thanks, this will definitely help me get more testing done!
Quoted 6 lines
Quoted 3 lines
I did some quick tests with some io benchmarks and found in a simple scenario that the scheduler worked as expected, except that it took very long. I'll investigate further and revert back.
Thanks. I will wait for details.
I'll try and send something out by Friday, but for now I am not even very sure if it is a real problem. I ran iozone on two groups with 500 and 1000 as weights on the same parition and set fairness to 1 in sysfs for the partition. I used a record size of 4 (default) and tried to run it on a file size of 1G.
BTW, I don't see anything about weights being multiple of an expected figure documented anywhere. I tried weights of 1024 (similar to the scheduler and got shouted back at :) ). Does the documentation patch specify the expected range for weights?
Signature · 1 line
Balbir
Quoted 43 lines · Balbir Singh
Quoted 28 lines · Vivek Goyal
Quoted 19 lines · Balbir Singh
Quoted 7 lines · Vivek Goyal
Hi All,
Here is the V6 of the IO controller patches generated on top of 2.6.31-rc1.
Previous versions of the patches was posted here.
Quoted 9 lines
(V1) http://lkml.org/lkml/2009/3/11/486
(V2) http://lkml.org/lkml/2009/5/5/275
(V3) http://lkml.org/lkml/2009/5/26/472
(V4) http://lkml.org/lkml/2009/6/8/580
(V5) http://lkml.org/lkml/2009/6/19/279This patchset is still work in progress but I want to keep on getting the snapshot of my tree out at regular intervals to get the feedback hence V6.
Quoted 7 lines
Hi, Vivek,
I was able to compile and boot a 2.6.31-rc1 kernel with this patchset. I have a request could you fold up all patches and make one consolidated patch available somewhere (makes it easier to test), may be a git tree?
Quoted 6 lines
Thanks for trying it out balbir. Ok, for ease of patching and testing, I will also maintain a consolidated patch. For V6 you can download the patch from here.
http://people.redhat.com/~vgoyal/io-controller/io-scheduler-based-io-controller-v6.patch
Thanks, this will definitely help me get more testing done!
Quoted 4 lines
Quoted 3 lines
I did some quick tests with some io benchmarks and found in a simple scenario that the scheduler worked as expected, except that it took very long. I'll investigate further and revert back.
Quoted 9 lines
Quoted 2 lines
Thanks. I will wait for details.
I'll try and send something out by Friday, but for now I am not even very sure if it is a real problem. I ran iozone on two groups with 500 and 1000 as weights on the same parition and set fairness to 1 in sysfs for the partition. I used a record size of 4 (default) and tried to run it on a file size of 1G.
Hi Balbir,
Trying iozone might be a good idea for analyzing the performance impact of io controller patches but it might not be the best thing to test fairness.
The biggest reason being that IO controller provides fairness only if there is constant contention between the groups. If one group goes away for sometime, other gets to use the disk full. Now while running above benchmark, there are numerous occasions where disk is not contended for and we don't see fairness numbers in user space.
I would recommend trying out fio or small tests to begin with which can create continuously backlogged queues at the disk to see how accurate the io-controller is.
Quoted 5 lines
BTW, I don't see anything about weights being multiple of an expected figure documented anywhere. I tried weights of 1024 (similar to the scheduler and got shouted back at :) ). Does the documentation patch specify the expected range for weights?
Weight range is 1-1000. I will update documentation to reflect this. Thanks for pointing it out.
Vivek
A message that is not in the archive (sent off-list or not yet received): 20090708143925.GE3215-SINUvgVNF2CyUtPGxGje5AC/[email protected]
Quoted 43 lines · Balbir Singh
Quoted 28 lines · Vivek Goyal
Quoted 19 lines · Balbir Singh
Quoted 7 lines · Vivek Goyal
Hi All,
Here is the V6 of the IO controller patches generated on top of 2.6.31-rc1.
Previous versions of the patches was posted here.
Quoted 9 lines
(V1) http://lkml.org/lkml/2009/3/11/486
(V2) http://lkml.org/lkml/2009/5/5/275
(V3) http://lkml.org/lkml/2009/5/26/472
(V4) http://lkml.org/lkml/2009/6/8/580
(V5) http://lkml.org/lkml/2009/6/19/279This patchset is still work in progress but I want to keep on getting the snapshot of my tree out at regular intervals to get the feedback hence V6.
Quoted 7 lines
Hi, Vivek,
I was able to compile and boot a 2.6.31-rc1 kernel with this patchset. I have a request could you fold up all patches and make one consolidated patch available somewhere (makes it easier to test), may be a git tree?
Quoted 6 lines
Thanks for trying it out balbir. Ok, for ease of patching and testing, I will also maintain a consolidated patch. For V6 you can download the patch from here.
http://people.redhat.com/~vgoyal/io-controller/io-scheduler-based-io-controller-v6.patch
Thanks, this will definitely help me get more testing done!
Quoted 4 lines
Quoted 3 lines
I did some quick tests with some io benchmarks and found in a simple scenario that the scheduler worked as expected, except that it took very long. I'll investigate further and revert back.
Quoted 9 lines
Quoted 2 lines
Thanks. I will wait for details.
I'll try and send something out by Friday, but for now I am not even very sure if it is a real problem. I ran iozone on two groups with 500 and 1000 as weights on the same parition and set fairness to 1 in sysfs for the partition. I used a record size of 4 (default) and tried to run it on a file size of 1G.
Hi Balbir,
Trying iozone might be a good idea for analyzing the performance impact of io controller patches but it might not be the best thing to test fairness.
The biggest reason being that IO controller provides fairness only if there is constant contention between the groups. If one group goes away for sometime, other gets to use the disk full. Now while running above benchmark, there are numerous occasions where disk is not contended for and we don't see fairness numbers in user space.
I would recommend trying out fio or small tests to begin with which can create continuously backlogged queues at the disk to see how accurate the io-controller is.
Quoted 5 lines
BTW, I don't see anything about weights being multiple of an expected figure documented anywhere. I tried weights of 1024 (similar to the scheduler and got shouted back at :) ). Does the documentation patch specify the expected range for weights?
Weight range is 1-1000. I will update documentation to reflect this. Thanks for pointing it out.
Vivek
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 28 lines · Vivek Goyal
Quoted 19 lines · Balbir Singh
Quoted 7 lines · Vivek Goyal
Hi All,
Here is the V6 of the IO controller patches generated on top of 2.6.31-rc1.
Previous versions of the patches was posted here.
Quoted 9 lines
(V1) http://lkml.org/lkml/2009/3/11/486
(V2) http://lkml.org/lkml/2009/5/5/275
(V3) http://lkml.org/lkml/2009/5/26/472
(V4) http://lkml.org/lkml/2009/6/8/580
(V5) http://lkml.org/lkml/2009/6/19/279This patchset is still work in progress but I want to keep on getting the snapshot of my tree out at regular intervals to get the feedback hence V6.
Quoted 7 lines
Hi, Vivek,
I was able to compile and boot a 2.6.31-rc1 kernel with this patchset. I have a request could you fold up all patches and make one consolidated patch available somewhere (makes it easier to test), may be a git tree?
Quoted 6 lines
Thanks for trying it out balbir. Ok, for ease of patching and testing, I will also maintain a consolidated patch. For V6 you can download the patch from here.
http://people.redhat.com/~vgoyal/io-controller/io-scheduler-based-io-controller-v6.patch
Thanks, this will definitely help me get more testing done!
Quoted 6 lines
Quoted 3 lines
I did some quick tests with some io benchmarks and found in a simple scenario that the scheduler worked as expected, except that it took very long. I'll investigate further and revert back.
Thanks. I will wait for details.
I'll try and send something out by Friday, but for now I am not even very sure if it is a real problem. I ran iozone on two groups with 500 and 1000 as weights on the same parition and set fairness to 1 in sysfs for the partition. I used a record size of 4 (default) and tried to run it on a file size of 1G.
BTW, I don't see anything about weights being multiple of an expected figure documented anywhere. I tried weights of 1024 (similar to the scheduler and got shouted back at :) ). Does the documentation patch specify the expected range for weights?
Signature · 1 line
Balbir
A message that is not in the archive (sent off-list or not yet received): 20090708035621.GB3215-SINUvgVNF2CyUtPGxGje5AC/[email protected]
Quoted 19 lines · Balbir Singh
Quoted 7 lines · Vivek Goyal
Hi All,
Here is the V6 of the IO controller patches generated on top of 2.6.31-rc1.
Previous versions of the patches was posted here.
Quoted 9 lines
(V1) http://lkml.org/lkml/2009/3/11/486
(V2) http://lkml.org/lkml/2009/5/5/275
(V3) http://lkml.org/lkml/2009/5/26/472
(V4) http://lkml.org/lkml/2009/6/8/580
(V5) http://lkml.org/lkml/2009/6/19/279This patchset is still work in progress but I want to keep on getting the snapshot of my tree out at regular intervals to get the feedback hence V6.
Quoted 7 lines
Hi, Vivek,
I was able to compile and boot a 2.6.31-rc1 kernel with this patchset. I have a request could you fold up all patches and make one consolidated patch available somewhere (makes it easier to test), may be a git tree?
Thanks for trying it out balbir. Ok, for ease of patching and testing, I will also maintain a consolidated patch. For V6 you can download the patch from here.
http://people.redhat.com/~vgoyal/io-controller/io-scheduler-based-io-controller-v6.patch
Quoted 3 lines
I did some quick tests with some io benchmarks and found in a simple scenario that the scheduler worked as expected, except that it took very long. I'll investigate further and revert back.
Thanks. I will wait for details.
Vivek
PATCH 1/1 io-controller: implement per group request allocation limitation
In reply to Vivek GoyalHi Vivek,
This patch exports a cgroup based per group request limits interface. and removes the global one. Now we can use this interface to perform different request allocation limitation for different groups.
--- block/blk-core.c | 23 ++++++++++-- block/blk-settings.c | 1 - block/blk-sysfs.c | 43 ----------------------- block/elevator-fq.c | 94 ++++++++++++++++++++++++++++++++++++++++++++++--- block/elevator-fq.h | 4 ++ 5 files changed, 111 insertions(+), 54 deletions(-)
block/blk-core.c +19 -4
@@ -722,13 +722,20 @@ static void ioc_set_batching(struct request_queue *q, struct io_context *ioc)static void __freed_request (struct request_queue *q,int sync,struct request_list *rl) { +struct io_group *iog; +unsigned long nr_group_requests; +if (q->rq_data.count[sync] <queue_congestion_off_threshold (q))blk_clear_queue_congested (q, sync);if (q->rq_data.count[sync] +1 <= q->nr_requests)blk_clear_queue_full (q, sync); -if (rl->count[sync] +1 <= q->nr_group_requests) { + iog =rl_iog (rl); + + nr_group_requests =get_group_requests (q, iog); + +if (nr_group_requests && rl->count[sync] +1 <= nr_group_requests) {if (waitqueue_active (&rl->wait[sync]))wake_up (&rl->wait[sync]); }
@@ -828,6 +835,8 @@ static struct request *get_request(struct request_queue *q, int rw_flags,const bool is_sync =rw_is_sync (rw_flags) !=0 ;int may_queue, priv;int sleep_on_global =0 ; +struct io_group *iog; +unsigned long nr_group_requests; may_queue =elv_may_queue (q, rw_flags);if (may_queue ==ELV_MQUEUE_NO )
@@ -843,7 +852,12 @@ static struct request *get_request(struct request_queue *q, int rw_flags,if (q->rq_data.count[is_sync]+1 >= q->nr_requests)blk_set_queue_full (q, is_sync); -if (rl->count[is_sync]+1 >= q->nr_group_requests) { + iog =rl_iog (rl); + + nr_group_requests =get_group_requests (q, iog); + +if (nr_group_requests && + rl->count[is_sync]+1 >= nr_group_requests) { ioc =current_io_context (GFP_ATOMIC , q->node);/* * The queue request descriptor group will fill after this
@@ -852,7 +866,7 @@ static struct request *get_request(struct request_queue *q, int rw_flags,* This process will be allowed to complete a batch of * requests, others will be blocked. */ -if (rl->count[is_sync] <= q->nr_group_requests) +if (rl->count[is_sync] <= nr_group_requests)ioc_set_batching (q, ioc);else { if (may_queue != ELV_MQUEUE_MUST
@@ -898,7 +912,8 @@ static struct request *get_request(struct request_queue *q, int rw_flags,* from per group request list */ -if (rl->count[is_sync] >= (3 * q->nr_group_requests /2 )) +if (nr_group_requests && + rl->count[is_sync] >= (3 * nr_group_requests /2 )) goto out; rl->starved[is_sync] =0 ;
block/blk-settings.c +0 -1
@@ -148,7 +148,6 @@ void blk_queue_make_request(struct request_queue *q, make_request_fn *mfn)* set defaults */ q->nr_requests =BLKDEV_MAX_RQ ; - q->nr_group_requests =BLKDEV_MAX_GROUP_RQ ; q->make_request_fn = mfn;blk_queue_dma_alignment (q,511 );
block/blk-sysfs.c +0 -43
63 lines not on this page · raw message
block/elevator-fq.c +88 -6
142 lines not on this page · raw message
block/elevator-fq.h +4 -0
@@ -308,6 +308,7 @@ struct io_cgroup {unsigned int weight;unsigned short ioprio_class; +unsigned long nr_group_requests;/* list of io_policy_node */ struct list_head policy_list;
@@ -386,6 +387,9 @@ struct elv_fq_data {unsigned int fairness; }; +extern unsigned short get_group_requests (struct request_queue *q, +struct io_group *iog); +/* Logging facilities. */ #ifdef CONFIG_DEBUG_GROUP_IOSCHED #define elv_log_ioq(efqd, ioq, fmt, args...) \
Signature · 1 line
1.5.4.rc3
Re: [PATCH] io-controller: implement per group request allocation limitation
Quoted 6 lines · Gui Jianfeng
Hi Vivek,
This patch exports a cgroup based per group request limits interface. and removes the global one. Now we can use this interface to perform different request allocation limitation for different groups.
Thanks Gui. Few points come to mind.
- You seem to be making this as per cgroup limit on all devices. I guess that different devices in the system can have different settings of q->nr_requests and hence will probably want different per group limit. So we might have to make it per cgroup per device limit.
- There does not seem to be any checks for making sure that children cgroups don't have more request descriptors allocated than parent group.
- I am re-thinking that what's the advantage of configuring request descriptors also through cgroups. It does bring in additional complexity with it and it should justfiy the advantages. Can you think of some?
Until and unless we can come up with some significant advantages, I will prefer to continue to use per group limit through q->nr_group_requests interface instead of cgroup. Once things stablize, we can revisit it and see how this interface can be improved.
Thanks
Vivek
Quoted 336 lines
Signed-off-by Gui Jianfeng <[email protected]>--- block/blk-core.c | 23 ++++++++++-- block/blk-settings.c | 1 - block/blk-sysfs.c | 43 ----------------------- block/elevator-fq.c | 94 ++++++++++++++++++++++++++++++++++++++++++++++--- block/elevator-fq.h | 4 ++ 5 files changed, 111 insertions(+), 54 deletions(-)block/blk-core.c +19
-4@@ -722,13 +722,20 @@ static void ioc_set_batching(struct request_queue *q, struct io_context *ioc)static void __freed_request (struct request_queue *q,int sync,struct request_list *rl) { +struct io_group *iog; +unsigned long nr_group_requests; +if (q->rq_data.count[sync] <queue_congestion_off_threshold (q))blk_clear_queue_congested (q, sync);if (q->rq_data.count[sync] +1 <= q->nr_requests)blk_clear_queue_full (q, sync); -if (rl->count[sync] +1 <= q->nr_group_requests) { + iog =rl_iog (rl); + + nr_group_requests =get_group_requests (q, iog); + +if (nr_group_requests && rl->count[sync] +1 <= nr_group_requests) {if (waitqueue_active (&rl->wait[sync]))wake_up (&rl->wait[sync]); }@@ -828,6 +835,8 @@ static struct request *get_request(struct request_queue *q, int rw_flags,const bool is_sync =rw_is_sync (rw_flags) !=0 ;int may_queue, priv;int sleep_on_global =0 ; +struct io_group *iog; +unsigned long nr_group_requests; may_queue =elv_may_queue (q, rw_flags);if (may_queue ==ELV_MQUEUE_NO )@@ -843,7 +852,12 @@ static struct request *get_request(struct request_queue *q, int rw_flags,if (q->rq_data.count[is_sync]+1 >= q->nr_requests)blk_set_queue_full (q, is_sync); -if (rl->count[is_sync]+1 >= q->nr_group_requests) { + iog =rl_iog (rl); + + nr_group_requests =get_group_requests (q, iog); + +if (nr_group_requests && + rl->count[is_sync]+1 >= nr_group_requests) { ioc =current_io_context (GFP_ATOMIC , q->node);/* * The queue request descriptor group will fill after this @@ -852,7 +866,7 @@ static struct request *get_request(struct request_queue *q, int rw_flags,* This process will be allowed to complete a batch of * requests, others will be blocked. */ -if (rl->count[is_sync] <= q->nr_group_requests) +if (rl->count[is_sync] <= nr_group_requests)ioc_set_batching (q, ioc);else { if (may_queue != ELV_MQUEUE_MUST@@ -898,7 +912,8 @@ static struct request *get_request(struct request_queue *q, int rw_flags,* from per group request list */ -if (rl->count[is_sync] >= (3 * q->nr_group_requests /2 )) +if (nr_group_requests && + rl->count[is_sync] >= (3 * nr_group_requests /2 )) goto out; rl->starved[is_sync] =0 ;block/blk-settings.c +0
-1@@ -148,7 +148,6 @@ void blk_queue_make_request(struct request_queue *q, make_request_fn *mfn)* set defaults */ q->nr_requests =BLKDEV_MAX_RQ ; - q->nr_group_requests =BLKDEV_MAX_GROUP_RQ ; q->make_request_fn = mfn;blk_queue_dma_alignment (q,511 );block/blk-sysfs.c +0
-43@@ -78,40 +78,8 @@ queue_requests_store(struct request_queue *q, const char *page, size_t count)return ret; }#ifdef CONFIG_GROUP_IOSCHED -static ssize_t queue_group_requests_show (struct request_queue *q,char *page) -{ -return queue_var_show (q->nr_group_requests, (page)); -} -extern void elv_io_group_congestion_threshold (struct request_queue *q,struct io_group *iog); - -static ssize_t -queue_group_requests_store (struct request_queue *q,const char *page, -size_t count) -{ -struct hlist_node *n; -struct io_group *iog; -struct elv_fq_data *efqd; -unsigned long nr; -int ret =queue_var_store (&nr, page, count); - -if (nr <BLKDEV_MIN_RQ ) - nr =BLKDEV_MIN_RQ ; - -spin_lock_irq (q->queue_lock); - - q->nr_group_requests = nr; - - efqd = &q->elevator->efqd; - -hlist_for_each_entry (iog, n, &efqd->group_list, elv_data_node) { -elv_io_group_congestion_threshold (q, iog); - } - -spin_unlock_irq (q->queue_lock); -return ret; -}#endif static ssize_t queue_ra_show (struct request_queue *q,char *page)@@ -278,14 +246,6 @@ static struct queue_sysfs_entry queue_requests_entry = { .store = queue_requests_store, }; -#ifdef CONFIG_GROUP_IOSCHED -static struct queue_sysfs_entry queue_group_requests_entry = { - .attr = {.name ="nr_group_requests" , .mode =S_IRUGO |S_IWUSR }, - .show = queue_group_requests_show, - .store = queue_group_requests_store, -}; -#endif -static struct queue_sysfs_entry queue_ra_entry = { .attr = {.name ="read_ahead_kb" , .mode =S_IRUGO |S_IWUSR }, .show = queue_ra_show,@@ -360,9 +320,6 @@ static struct queue_sysfs_entry queue_iostats_entry = {static struct attribute *default_attrs[] = { &queue_requests_entry.attr, -#ifdef CONFIG_GROUP_IOSCHED - &queue_group_requests_entry.attr, -#endif &queue_ra_entry.attr, &queue_max_hw_sectors_entry.attr, &queue_max_sectors_entry.attr,block/elevator-fq.c +88
-6@@ -59,6 +59,35 @@ elv_release_ioq(struct elevator_queue *eq, struct io_queue **ioq_ptr);#define for_each_entity_safe (entity, parent) \ for (; entity && ({ parent = entity->parent; 1; }); entity = parent) +unsigned short get_group_requests (struct request_queue *q, +struct io_group *iog) +{ +struct cgroup_subsys_state *css; +struct io_cgroup *iocg; +unsigned long nr_group_requests; + +if (!iog) +return q->nr_requests; + +rcu_read_lock (); + +if (!iog->iocg_id) { + nr_group_requests =0 ; + goto out; + } + + css =css_lookup (&io_subsys, iog->iocg_id); +if (!css) { + nr_group_requests =0 ; + goto out; + } + + iocg = container_of(css, struct io_cgroup, css); + nr_group_requests = iocg->nr_group_requests; +out: +rcu_read_unlock (); +return nr_group_requests; +}static struct io_entity *bfq_lookup_next_entity (struct io_sched_data *sd,int extract);@@ -1257,14 +1286,17 @@ void elv_io_group_congestion_threshold(struct request_queue *q, struct io_group *iog) {int nr; +unsigned long nr_group_requests; - nr = q->nr_group_requests - (q->nr_group_requests /8 ) +1 ; -if (nr > q->nr_group_requests) - nr = q->nr_group_requests; + nr_group_requests =get_group_requests (q, iog); + + nr = nr_group_requests - (nr_group_requests /8 ) +1 ; +if (nr > nr_group_requests) + nr = nr_group_requests; iog->nr_congestion_on = nr; - nr = q->nr_group_requests - (q->nr_group_requests /8 ) - - (q->nr_group_requests /16 ) -1 ; + nr = nr_group_requests - (nr_group_requests /8 ) + - (nr_group_requests /16 ) -1 ;if (nr <1 ) nr =1 ; iog->nr_congestion_off = nr;@@ -1283,6 +1315,7 @@ int elv_io_group_congested(struct request_queue *q, struct page *page, int sync) {struct io_group *iog;int ret =0 ; +unsigned long nr_group_requests;rcu_read_lock ();@@ -1300,10 +1333,11 @@ int elv_io_group_congested(struct request_queue *q, struct page *page, int sync) } ret =elv_is_iog_congested (q, iog, sync); + nr_group_requests =get_group_requests (q, iog);if (ret)elv_log_iog (&q->elevator->efqd, iog,"iog congested=%d sync=%d" " rl.count[sync]=%d nr_group_requests=%d" , - ret, sync, iog->rl.count[sync], q->nr_group_requests); + ret, sync, iog->rl.count[sync], nr_group_requests);rcu_read_unlock ();return ret; }@@ -1549,6 +1583,48 @@ free_buf:return ret; } +static u64 io_cgroup_nr_requests_read (struct cgroup *cgroup, +struct cftype *cftype) +{ +struct io_cgroup *iocg; +u64 ret; + +if (!cgroup_lock_live_group (cgroup)) +return -ENODEV ; + + iocg =cgroup_to_io_cgroup (cgroup); +spin_lock_irq (&iocg->lock); + ret = iocg->nr_group_requests; +spin_unlock_irq (&iocg->lock); + +cgroup_unlock (); + +return ret; +} + +static int io_cgroup_nr_requests_write (struct cgroup *cgroup, +struct cftype *cftype, +u64 val) +{ +struct io_cgroup *iocg; + +if (val <BLKDEV_MIN_RQ ) + val =BLKDEV_MIN_RQ ; + +if (!cgroup_lock_live_group (cgroup)) +return -ENODEV ; + + iocg =cgroup_to_io_cgroup (cgroup); + +spin_lock_irq (&iocg->lock); + iocg->nr_group_requests = (unsigned long )val; +spin_unlock_irq (&iocg->lock); + +cgroup_unlock (); + +return 0 ; +} +#define SHOW_FUNCTION (__VAR) \ static u64 io_cgroup_##__VAR##_read(struct cgroup *cgroup, \ struct cftype *cftype) \@@ -1735,6 +1811,11 @@ static int io_cgroup_disk_dequeue_read(struct cgroup *cgroup,struct cftype bfqio_files[] = { { + .name ="nr_group_requests" , + .read_u64 = io_cgroup_nr_requests_read, + .write_u64 = io_cgroup_nr_requests_write, + }, + { .name ="policy" , .read_seq_string = io_cgroup_policy_read, .write_string = io_cgroup_policy_write,@@ -1790,6 +1871,7 @@ static struct cgroup_subsys_state *iocg_create(struct cgroup_subsys *subsys,spin_lock_init (&iocg->lock);INIT_HLIST_HEAD (&iocg->group_data); + iocg->nr_group_requests =BLKDEV_MAX_GROUP_RQ ; iocg->weight =IO_DEFAULT_GRP_WEIGHT ; iocg->ioprio_class =IO_DEFAULT_GRP_CLASS ;INIT_LIST_HEAD (&iocg->policy_list);block/elevator-fq.h +4
-0@@ -308,6 +308,7 @@ struct io_cgroup {unsigned int weight;unsigned short ioprio_class; +unsigned long nr_group_requests;/* list of io_policy_node */ struct list_head policy_list;@@ -386,6 +387,9 @@ struct elv_fq_data {unsigned int fairness; }; +extern unsigned short get_group_requests (struct request_queue *q, +struct io_group *iog); +/* Logging facilities. */ #ifdef CONFIG_DEBUG_GROUP_IOSCHED #define elv_log_ioq(efqd, ioq, fmt, args...) \Signature · 1 line
1.5.4.rc3
Re: [PATCH] io-controller: implement per group request allocation limitation
Vivek Goyal wrote, on 07/13/2009 12:03 PM:
Quoted 8 lines · Gui Jianfeng
Quoted 6 lines
Hi Vivek,
This patch exports a cgroup based per group request limits interface. and removes the global one. Now we can use this interface to perform different request allocation limitation for different groups.
Quoted 6 lines
Thanks Gui. Few points come to mind.
- You seem to be making this as per cgroup limit on all devices. I guess that different devices in the system can have different settings of
q->nr_requestsand hence will probably want different per group limit. So we might have to make it per cgroup per device limit.
From the viewpoint of implementation, there is a difficulty in my mind to implement per cgroup per device limit arising from that io_group is allocated when associated device is firstly used. I guess Gui chose per cgroup limit on all devices approach because of this, right?
Quoted 11 lines
- There does not seem to be any checks for making sure that children cgroups don't have more request descriptors allocated than parent group.
- I am re-thinking that what's the advantage of configuring request descriptors also through cgroups. It does bring in additional complexity with it and it should justfiy the advantages. Can you think of some?
Until and unless we can come up with some significant advantages, I will prefer to continue to use per group limit through
q->nr_group_requestsinterface instead of cgroup. Once things stablize, we can revisit it and see how this interface can be improved.
I agree. I will try to clarify if per group per device limitation is needed or not (or, if it has the advantage beyond the complexity) through some tests.
Tnaks a lot,
Muuhh
Quoted 340 lines
Thanks
VivekQuoted 336 lines
Signed-off-by Gui Jianfeng<[email protected]>--- block/blk-core.c | 23 ++++++++++-- block/blk-settings.c | 1 - block/blk-sysfs.c | 43 ----------------------- block/elevator-fq.c | 94 ++++++++++++++++++++++++++++++++++++++++++++++--- block/elevator-fq.h | 4 ++ 5 files changed, 111 insertions(+), 54 deletions(-)block/blk-core.c +19
-4@@ -722,13 +722,20 @@ static void ioc_set_batching(struct request_queue *q, struct io_context *ioc)static void __freed_request (struct request_queue *q,int sync,struct request_list *rl) { +struct io_group *iog; +unsigned long nr_group_requests; +if (q->rq_data.count[sync]<queue_congestion_off_threshold (q))blk_clear_queue_congested (q, sync);if (q->rq_data.count[sync] +1 <= q->nr_requests)blk_clear_queue_full (q, sync); -if (rl->count[sync] +1 <= q->nr_group_requests) { + iog =rl_iog (rl); + + nr_group_requests =get_group_requests (q, iog); + +if (nr_group_requests&& rl->count[sync] +1 <= nr_group_requests) {if (waitqueue_active (&rl->wait[sync]))wake_up (&rl->wait[sync]); }@@ -828,6 +835,8 @@ static struct request *get_request(struct request_queue *q, int rw_flags,const bool is_sync =rw_is_sync (rw_flags) !=0 ;int may_queue, priv;int sleep_on_global =0 ; +struct io_group *iog; +unsigned long nr_group_requests; may_queue =elv_may_queue (q, rw_flags);if (may_queue ==ELV_MQUEUE_NO )@@ -843,7 +852,12 @@ static struct request *get_request(struct request_queue *q, int rw_flags,if (q->rq_data.count[is_sync]+1 >= q->nr_requests)blk_set_queue_full (q, is_sync); -if (rl->count[is_sync]+1 >= q->nr_group_requests) { + iog =rl_iog (rl); + + nr_group_requests =get_group_requests (q, iog); + +if (nr_group_requests&& + rl->count[is_sync]+1 >= nr_group_requests) { ioc =current_io_context (GFP_ATOMIC , q->node);/* * The queue request descriptor group will fill after this @@ -852,7 +866,7 @@ static struct request *get_request(struct request_queue *q, int rw_flags,* This process will be allowed to complete a batch of * requests, others will be blocked. */ -if (rl->count[is_sync]<= q->nr_group_requests) +if (rl->count[is_sync]<= nr_group_requests)ioc_set_batching (q, ioc);else { if (may_queue != ELV_MQUEUE_MUST@@ -898,7 +912,8 @@ static struct request *get_request(struct request_queue *q, int rw_flags,* from per group request list */ -if (rl->count[is_sync]>= (3 * q->nr_group_requests /2 )) +if (nr_group_requests&& + rl->count[is_sync]>= (3 * nr_group_requests /2 )) goto out; rl->starved[is_sync] =0 ;block/blk-settings.c +0
-1@@ -148,7 +148,6 @@ void blk_queue_make_request(struct request_queue *q, make_request_fn *mfn)* set defaults */ q->nr_requests =BLKDEV_MAX_RQ ; - q->nr_group_requests =BLKDEV_MAX_GROUP_RQ ; q->make_request_fn = mfn;blk_queue_dma_alignment (q,511 );block/blk-sysfs.c +0
-43@@ -78,40 +78,8 @@ queue_requests_store(struct request_queue *q, const char *page, size_t count)return ret; }#ifdef CONFIG_GROUP_IOSCHED -static ssize_t queue_group_requests_show (struct request_queue *q,char *page) -{ -return queue_var_show (q->nr_group_requests, (page)); -} -extern void elv_io_group_congestion_threshold (struct request_queue *q,struct io_group *iog); - -static ssize_t -queue_group_requests_store (struct request_queue *q,const char *page, -size_t count) -{ -struct hlist_node *n; -struct io_group *iog; -struct elv_fq_data *efqd; -unsigned long nr; -int ret =queue_var_store (&nr, page, count); - -if (nr<BLKDEV_MIN_RQ ) - nr =BLKDEV_MIN_RQ ; - -spin_lock_irq (q->queue_lock); - - q->nr_group_requests = nr; - - efqd =&q->elevator->efqd; - -hlist_for_each_entry (iog, n,&efqd->group_list, elv_data_node) { -elv_io_group_congestion_threshold (q, iog); - } - -spin_unlock_irq (q->queue_lock); -return ret; -}#endif static ssize_t queue_ra_show (struct request_queue *q,char *page)@@ -278,14 +246,6 @@ static struct queue_sysfs_entry queue_requests_entry = { .store = queue_requests_store, }; -#ifdef CONFIG_GROUP_IOSCHED -static struct queue_sysfs_entry queue_group_requests_entry = { - .attr = {.name ="nr_group_requests" , .mode =S_IRUGO |S_IWUSR }, - .show = queue_group_requests_show, - .store = queue_group_requests_store, -}; -#endif -static struct queue_sysfs_entry queue_ra_entry = { .attr = {.name ="read_ahead_kb" , .mode =S_IRUGO |S_IWUSR }, .show = queue_ra_show,@@ -360,9 +320,6 @@ static struct queue_sysfs_entry queue_iostats_entry = {static struct attribute *default_attrs[] = { &queue_requests_entry.attr, -#ifdef CONFIG_GROUP_IOSCHED - &queue_group_requests_entry.attr, -#endif &queue_ra_entry.attr, &queue_max_hw_sectors_entry.attr, &queue_max_sectors_entry.attr,block/elevator-fq.c +88
-6@@ -59,6 +59,35 @@ elv_release_ioq(struct elevator_queue *eq, struct io_queue **ioq_ptr);#define for_each_entity_safe (entity, parent) \ for (; entity&& ({ parent = entity->parent; 1; }); entity = parent) +unsigned short get_group_requests (struct request_queue *q, +struct io_group *iog) +{ +struct cgroup_subsys_state *css; +struct io_cgroup *iocg; +unsigned long nr_group_requests; + +if (!iog) +return q->nr_requests; + +rcu_read_lock (); + +if (!iog->iocg_id) { + nr_group_requests =0 ; + goto out; + } + + css =css_lookup (&io_subsys, iog->iocg_id); +if (!css) { + nr_group_requests =0 ; + goto out; + } + + iocg = container_of(css, struct io_cgroup, css); + nr_group_requests = iocg->nr_group_requests; +out: +rcu_read_unlock (); +return nr_group_requests; +}static struct io_entity *bfq_lookup_next_entity (struct io_sched_data *sd,int extract);@@ -1257,14 +1286,17 @@ void elv_io_group_congestion_threshold(struct request_queue *q, struct io_group *iog) {int nr; +unsigned long nr_group_requests; - nr = q->nr_group_requests - (q->nr_group_requests /8 ) +1 ; -if (nr> q->nr_group_requests) - nr = q->nr_group_requests; + nr_group_requests =get_group_requests (q, iog); + + nr = nr_group_requests - (nr_group_requests /8 ) +1 ; +if (nr> nr_group_requests) + nr = nr_group_requests; iog->nr_congestion_on = nr; - nr = q->nr_group_requests - (q->nr_group_requests /8 ) - - (q->nr_group_requests /16 ) -1 ; + nr = nr_group_requests - (nr_group_requests /8 ) + - (nr_group_requests /16 ) -1 ;if (nr<1 ) nr =1 ; iog->nr_congestion_off = nr;@@ -1283,6 +1315,7 @@ int elv_io_group_congested(struct request_queue *q, struct page *page, int sync) {struct io_group *iog;int ret =0 ; +unsigned long nr_group_requests;rcu_read_lock ();@@ -1300,10 +1333,11 @@ int elv_io_group_congested(struct request_queue *q, struct page *page, int sync) } ret =elv_is_iog_congested (q, iog, sync); + nr_group_requests =get_group_requests (q, iog);if (ret)elv_log_iog (&q->elevator->efqd, iog,"iog congested=%d sync=%d" " rl.count[sync]=%d nr_group_requests=%d" , - ret, sync, iog->rl.count[sync], q->nr_group_requests); + ret, sync, iog->rl.count[sync], nr_group_requests);rcu_read_unlock ();return ret; }@@ -1549,6 +1583,48 @@ free_buf:return ret; } +static u64 io_cgroup_nr_requests_read (struct cgroup *cgroup, +struct cftype *cftype) +{ +struct io_cgroup *iocg; +u64 ret; + +if (!cgroup_lock_live_group (cgroup)) +return -ENODEV ; + + iocg =cgroup_to_io_cgroup (cgroup); +spin_lock_irq (&iocg->lock); + ret = iocg->nr_group_requests; +spin_unlock_irq (&iocg->lock); + +cgroup_unlock (); + +return ret; +} + +static int io_cgroup_nr_requests_write (struct cgroup *cgroup, +struct cftype *cftype, +u64 val) +{ +struct io_cgroup *iocg; + +if (val<BLKDEV_MIN_RQ ) + val =BLKDEV_MIN_RQ ; + +if (!cgroup_lock_live_group (cgroup)) +return -ENODEV ; + + iocg =cgroup_to_io_cgroup (cgroup); + +spin_lock_irq (&iocg->lock); + iocg->nr_group_requests = (unsigned long )val; +spin_unlock_irq (&iocg->lock); + +cgroup_unlock (); + +return 0 ; +} +#define SHOW_FUNCTION (__VAR) \ static u64 io_cgroup_##__VAR##_read(struct cgroup *cgroup, \ struct cftype *cftype) \@@ -1735,6 +1811,11 @@ static int io_cgroup_disk_dequeue_read(struct cgroup *cgroup,struct cftype bfqio_files[] = { { + .name ="nr_group_requests" , + .read_u64 = io_cgroup_nr_requests_read, + .write_u64 = io_cgroup_nr_requests_write, + }, + { .name ="policy" , .read_seq_string = io_cgroup_policy_read, .write_string = io_cgroup_policy_write,@@ -1790,6 +1871,7 @@ static struct cgroup_subsys_state *iocg_create(struct cgroup_subsys *subsys,spin_lock_init (&iocg->lock);INIT_HLIST_HEAD (&iocg->group_data); + iocg->nr_group_requests =BLKDEV_MAX_GROUP_RQ ; iocg->weight =IO_DEFAULT_GRP_WEIGHT ; iocg->ioprio_class =IO_DEFAULT_GRP_CLASS ;INIT_LIST_HEAD (&iocg->policy_list);block/elevator-fq.h +4
-1@@ -308,6 +308,7 @@ struct io_cgroup {unsigned int weight;unsigned short ioprio_class; +unsigned long nr_group_requests;/* list of io_policy_node */ struct list_head policy_list;@@ -386,6 +387,9 @@ struct elv_fq_data {unsigned int fairness; }; +extern unsigned short get_group_requests (struct request_queue *q, +struct io_group *iog); +/* Logging facilities. */ #ifdef CONFIG_DEBUG_GROUP_IOSCHED #define elv_log_ioq(efqd, ioq, fmt, args...) \ --1.5.4.rc3
Signature · 3 lines
IKEDA, Munehiro
NEC Corporation of America
[email protected]Re: [PATCH] io-controller: implement per group request allocation limitation
Quoted 16 lines · Munehiro Ikeda
Vivek Goyal wrote, on 07/13/2009 12:03 PM:
Quoted 8 lines · Gui Jianfeng
Quoted 6 lines
Hi Vivek,
This patch exports a cgroup based per group request limits interface. and removes the global one. Now we can use this interface to perform different request allocation limitation for different groups.
Quoted 6 lines
Thanks Gui. Few points come to mind.
- You seem to be making this as per cgroup limit on all devices. I guess that different devices in the system can have different settings of
q->nr_requestsand hence will probably want different per group limit. So we might have to make it per cgroup per device limit.
Quoted 5 lines
From the viewpoint of implementation, there is a difficulty in my mind to implement per cgroup per device limit arising from that
io_groupis allocated
when associated device is firstly used. I guess Gui chose per cgroup limit on all devices approach because of this, right?
Yes, I choose this solution from the simplicity point of view, the code will get complicated if choosing per cgroup per device limit. But it seems per cgroup per device limits is more reasonable.
Quoted 18 lines
Quoted 11 lines
- There does not seem to be any checks for making sure that children cgroups don't have more request descriptors allocated than parent
group.- I am re-thinking that what's the advantage of configuring request descriptors also through cgroups. It does bring in additional
complexity
with it and it should justfiy the advantages. Can you think of some?Quoted 6 lines
Until and unless we can come up with some significant advantages, I
will
prefer to continue to use per group limit throughq->nr_group_requestsinterface instead of cgroup. Once things stablize, we can revisit
it and
see how this interface can be improved.
Quoted 4 lines
I agree. I will try to clarify if per group per device limitation is needed
or not (or, if it has the advantage beyond the complexity) through some tests.
Great, hope to hear you soon.
Signature · 2 lines
Regards Gui Jianfeng
Quoted 368 lines
Tnaks a lot,
MuuhhQuoted 360 lines
Thanks
VivekQuoted 356 lines
Signed-off-by Gui Jianfeng<[email protected]>--- block/blk-core.c | 23 ++++++++++-- block/blk-settings.c | 1 - block/blk-sysfs.c | 43 ----------------------- block/elevator-fq.c | 94 ++++++++++++++++++++++++++++++++++++++++++++++--- block/elevator-fq.h | 4 ++ 5 files changed, 111 insertions(+), 54 deletions(-)block/blk-core.c +8
-1@@ -722,13 +722,20 @@ static void ioc_set_batching(struct request_queue *q, struct io_context *ioc) static void __freed_request(struct request_queue *q, int sync, struct request_list *rl) { + struct io_group *iog; +unsigned long nr_group_requests; +if (q->rq_data.count[sync]<queue_congestion_off_threshold (q))blk_clear_queue_congested (q, sync);if (q->rq_data.count[sync] +1 <= q->nr_requests)blk_clear_queue_full (q, sync); - if (rl->count[sync] + 1<= q->nr_group_requests) { + iog =rl_iog (rl); + + nr_group_requests =get_group_requests (q, iog); + +if (nr_group_requests&& rl->count[sync] +1 <= nr_group_requests) {if (waitqueue_active (&rl->wait[sync]))wake_up (&rl->wait[sync]);}block/blk-core.c +2
-0@@ -828,6 +835,8 @@ static struct request *get_request(struct request_queue *q, int rw_flags, const bool is_sync = rw_is_sync(rw_flags) != 0;int may_queue, priv;int sleep_on_global =0 ; +struct io_group *iog; +unsigned long nr_group_requests; may_queue =elv_may_queue (q, rw_flags);if (may_queue == ELV_MQUEUE_NO)block/blk-core.c +6
-1@@ -843,7 +852,12 @@ static struct request *get_request(struct request_queue *q, int rw_flags, if (q->rq_data.count[is_sync]+1>= q->nr_requests) blk_set_queue_full(q, is_sync); -if (rl->count[is_sync]+1 >= q->nr_group_requests) { + iog = rl_iog(rl); + + nr_group_requests =get_group_requests (q, iog); + +if (nr_group_requests&& + rl->count[is_sync]+1 >= nr_group_requests) { ioc =current_io_context (GFP_ATOMIC , q->node);/* * The queue request descriptor group will fill after this
block/blk-core.c +1
-1@@ -852,7 +866,7 @@ static struct request *get_request(structrequest_queue *q, int rw_flags, * This process will be allowed to complete a batch of * requests, others will be blocked. */ -if (rl->count[is_sync]<= q->nr_group_requests) +if (rl->count[is_sync]<= nr_group_requests)ioc_set_batching (q, ioc);else {if (may_queue != ELV_MQUEUE_MUSTblock/blk-core.c +2
-1@@ -898,7 +912,8 @@ static struct request *get_request(structrequest_queue *q, int rw_flags, * from per group request list */ -if (rl->count[is_sync]>= (3 * q->nr_group_requests /2 )) +if (nr_group_requests&& + rl->count[is_sync]>= (3 * nr_group_requests /2 )) goto out;rl->starved[is_sync] = 0;block/blk-settings.c +0
-1@@ -148,7 +148,6 @@ void blk_queue_make_request(struct request_queue*q, make_request_fn *mfn) * set defaults */ q->nr_requests =BLKDEV_MAX_RQ ; - q->nr_group_requests =BLKDEV_MAX_GROUP_RQ ; q->make_request_fn = mfn;blk_queue_dma_alignment(q, 511);block/blk-sysfs.c +0
-32@@ -78,40 +78,8 @@ queue_requests_store(struct request_queue *q, const char *page, size_t count) return ret; }#ifdef CONFIG_GROUP_IOSCHED -static ssize_t queue_group_requests_show (struct request_queue *q, char *page) -{ -return queue_var_show (q->nr_group_requests, (page)); -} -extern void elv_io_group_congestion_threshold (struct request_queue *q,struct io_group *iog); - -static ssize_t -queue_group_requests_store (struct request_queue *q,const char *page, -size_t count) -{ -struct hlist_node *n; -struct io_group *iog; -struct elv_fq_data *efqd; -unsigned long nr; -int ret =queue_var_store (&nr, page, count); - -if (nr<BLKDEV_MIN_RQ ) - nr =BLKDEV_MIN_RQ ; - -spin_lock_irq (q->queue_lock); - - q->nr_group_requests = nr; - - efqd =&q->elevator->efqd; - -hlist_for_each_entry (iog, n,&efqd->group_list, elv_data_node) { -elv_io_group_congestion_threshold (q, iog); - } - -spin_unlock_irq (q->queue_lock); -return ret; -}#endif static
ssize_tqueue_ra_show(struct request_queue *q, char *page)block/blk-sysfs.c +0
-8@@ -278,14 +246,6 @@ static struct queue_sysfs_entry queue_requests_entry = { .store = queue_requests_store, }; -#ifdef CONFIG_GROUP_IOSCHED -static struct queue_sysfs_entry queue_group_requests_entry = { - .attr = {.name ="nr_group_requests" , .mode =S_IRUGO |S_IWUSR }, - .show = queue_group_requests_show, - .store = queue_group_requests_store, -}; -#endif -static struct queue_sysfs_entry queue_ra_entry = { .attr = {.name ="read_ahead_kb" , .mode =S_IRUGO |S_IWUSR },.show =
queue_ra_show,block/blk-sysfs.c +0
-3@@ -360,9 +320,6 @@ static struct queue_sysfs_entry queue_iostats_entry = {static struct attribute *default_attrs[] = { &queue_requests_entry.attr, -#ifdef CONFIG_GROUP_IOSCHED - &queue_group_requests_entry.attr, -#endif &queue_ra_entry.attr, &queue_max_hw_sectors_entry.attr,&queue_max_sectors_entry.attr,
block/elevator-fq.c +29
-0@@ -59,6 +59,35 @@ elv_release_ioq(struct elevator_queue *eq, struct io_queue **ioq_ptr);#define for_each_entity_safe (entity, parent) \ for (; entity&& ({ parent = entity->parent; 1; }); entity = parent) +unsigned short get_group_requests (struct request_queue *q, +struct io_group *iog) +{ +struct cgroup_subsys_state *css; +struct io_cgroup *iocg; +unsigned long nr_group_requests; + +if (!iog) +return q->nr_requests; + +rcu_read_lock (); + +if (!iog->iocg_id) { + nr_group_requests =0 ; + goto out; + } + + css =css_lookup (&io_subsys, iog->iocg_id); +if (!css) { + nr_group_requests =0 ; + goto out; + } + + iocg = container_of(css, struct io_cgroup, css); + nr_group_requests = iocg->nr_group_requests; +out: +rcu_read_unlock (); +return nr_group_requests; +}static struct
io_entity*bfq_lookup_next_entity(structio_sched_data*sd,int extract);block/elevator-fq.c +8
-5@@ -1257,14 +1286,17 @@ void elv_io_group_congestion_threshold(structrequest_queue *q, struct io_group *iog) { int nr; +unsigned long nr_group_requests; - nr = q->nr_group_requests - (q->nr_group_requests /8 ) +1 ; -if (nr> q->nr_group_requests) - nr = q->nr_group_requests; + nr_group_requests =get_group_requests (q, iog); + + nr = nr_group_requests - (nr_group_requests /8 ) +1 ; +if (nr> nr_group_requests) + nr = nr_group_requests; iog->nr_congestion_on = nr; - nr = q->nr_group_requests - (q->nr_group_requests /8 ) - - (q->nr_group_requests /16 ) -1 ; + nr = nr_group_requests - (nr_group_requests /8 ) + - (nr_group_requests /16 ) -1 ;if (nr<1 ) nr =1 ;iog->nr_congestion_off = nr;block/elevator-fq.c +2
-0@@ -1283,6 +1315,7 @@ int elv_io_group_congested(struct request_queue *q, struct page *page, int sync) {struct io_group *iog;int ret =0 ; +unsigned long nr_group_requests;rcu_read_lock ();@@ -1300,10 +1333,11 @@ int elv_io_group_congested(struct request_queue *q, struct page *page, int sync) } ret =elv_is_iog_congested (q, iog, sync); + nr_group_requests =get_group_requests (q, iog);if (ret) elv_log_iog(&q->elevator->efqd, iog, "iog congested=%dsync=%d"
" rl.count[sync]=%d nr_group_requests=%d",block/elevator-fq.c +48
-1- ret, sync, iog->rl.count[sync], q->nr_group_requests); + ret, sync, iog->rl.count[sync], nr_group_requests);rcu_read_unlock ();return ret; }@@ -1549,6 +1583,48 @@ free_buf:return ret; } +static u64 io_cgroup_nr_requests_read (struct cgroup *cgroup, +struct cftype *cftype) +{ +struct io_cgroup *iocg; +u64 ret; + +if (!cgroup_lock_live_group (cgroup)) +return -ENODEV ; + + iocg =cgroup_to_io_cgroup (cgroup); +spin_lock_irq (&iocg->lock); + ret = iocg->nr_group_requests; +spin_unlock_irq (&iocg->lock); + +cgroup_unlock (); + +return ret; +} + +static int io_cgroup_nr_requests_write (struct cgroup *cgroup, +struct cftype *cftype, +u64 val) +{ +struct io_cgroup *iocg; + +if (val<BLKDEV_MIN_RQ ) + val =BLKDEV_MIN_RQ ; + +if (!cgroup_lock_live_group (cgroup)) +return -ENODEV ; + + iocg =cgroup_to_io_cgroup (cgroup); + +spin_lock_irq (&iocg->lock); + iocg->nr_group_requests = (unsigned long )val; +spin_unlock_irq (&iocg->lock); + +cgroup_unlock (); + +return 0 ; +} +#define SHOW_FUNCTION (__VAR) \ static u64 io_cgroup_##__VAR##_read(struct cgroup *cgroup, \ struct cftype *cftype) \@@ -1735,6 +1811,11 @@ static int io_cgroup_disk_dequeue_read(structcgroup *cgroup, struct cftype bfqio_files[] = { { + .name ="nr_group_requests" , + .read_u64 = io_cgroup_nr_requests_read, + .write_u64 = io_cgroup_nr_requests_write, + }, + { .name ="policy" , .read_seq_string = io_cgroup_policy_read,.write_string =
io_cgroup_policy_write,block/elevator-fq.c +1
-0@@ -1790,6 +1871,7 @@ static struct cgroup_subsys_state *iocg_create(struct cgroup_subsys *subsys, spin_lock_init(&iocg->lock);INIT_HLIST_HEAD (&iocg->group_data); + iocg->nr_group_requests =BLKDEV_MAX_GROUP_RQ ; iocg->weight =IO_DEFAULT_GRP_WEIGHT ; iocg->ioprio_class =IO_DEFAULT_GRP_CLASS ;INIT_LIST_HEAD(&iocg->policy_list);block/elevator-fq.h +4
-0@@ -308,6 +308,7 @@ struct io_cgroup {unsigned int weight;unsigned short ioprio_class; +unsigned long nr_group_requests;/* list of io_policy_node */ struct list_head policy_list;@@ -386,6 +387,9 @@ struct elv_fq_data {unsigned int fairness; }; +extern unsigned short get_group_requests (struct request_queue *q, +struct io_group *iog); +/* Logging facilities. */ #ifdef CONFIG_DEBUG_GROUP_IOSCHED #define elv_log_ioq(efqd, ioq, fmt, args...) \Signature · 1 line
1.5.4.rc3
Re: [PATCH] io-controller: implement per group request allocation limitation
Gui Jianfeng wrote, on 07/14/2009 03:45 AM:
Quoted 41 lines
Quoted 14 lines · Munehiro Ikeda
Vivek Goyal wrote, on 07/13/2009 12:03 PM:
Quoted 7 lines · Gui Jianfeng
Quoted 6 lines
Hi Vivek,
This patch exports a cgroup based per group request limits interface. and removes the global one. Now we can use this interface to perform different request allocation limitation for different groups.
Quoted 6 lines
Thanks Gui. Few points come to mind.
- You seem to be making this as per cgroup limit on all devices. I guess that different devices in the system can have different settings of
q->nr_requestsand hence will probably want different per group limit. So we might have to make it per cgroup per device limit.Quoted 5 lines
From the viewpoint of implementation, there is a difficulty in my mind to
implement per cgroup per device limit arising from thatio_groupis allocated
when associated device is firstly used. I guess Gui chose per cgroup limit on all devices approach because of this, right?Yes, I choose this solution from the simplicity point of view, the code will get complicated if choosing per cgroup per device limit. But it seems per cgroup per device limits is more reasonable.
Quoted 16 lines
Quoted 16 lines
- There does not seem to be any checks for making sure that children cgroups don't have more request descriptors allocated than parent
group.- I am re-thinking that what's the advantage of configuring request descriptors also through cgroups. It does bring in additional
complexity
with it and it should justfiy the advantages. Can you think of some?Until and unless we can come up with some significant advantages, I
will
prefer to continue to use per group limit throughq->nr_group_requestsinterface instead of cgroup. Once things stablize, we can revisit
it and
see how this interface can be improved.
Quoted 6 lines
Quoted 4 lines
I agree. I will try to clarify if per group per device limitation is needed
or not (or, if it has the advantage beyond the complexity) through some tests.Great, hope to hear you soon.
Sorry for so late. I tried it, and write the result and my opinion below...
Scenario
====================
The possible scenario where per-cgroup nr_requests limitation is beneficial in my mind is that:
- Process P1 in cgroup "g1" is running with submitting many requests to a device. The number of the requests in the device queue is almost nr_requests for the device.
- After a while, process P2 in cgroup "g2" starts running. P2 also tries to submit requests as many as P1.
- Assuming that user wants P2 to grab bandwidth as soon as possible and keep it certain level.
In this scenario, I predicted the bandwidth behavior of P2 along with tuning global nr_group_requests like below.
- If having nr_group_requests almost same as nr_requests, P1 can allocate requests up to nr_requests and there is no room for P2 at the beginning of its running. As a result of it, P2 has to wait for a while till P1's requests are completed and rising of bandwidth is delayed.
- If having nr_group_requests fewer to restrict requests from P1 and make room for P2, the bandwidth of P2 may be lower than the case that P1 can allocate more requests.
If the prediction is correct and per-cgroup nr_requests limitation can make the situation better, per-cgroup nr_requests is supposed to be beneficial.
Verification Conditions
========================
- Kernel:
2.6.31-rc1
+ Patches from Vivek on Jul 2, 2009
(IO scheduler based IO controller V6)
https://lists.linux-foundation.org/pipermail/containers/2009-July/018948.html
+ Patches from Gui Jianfeng on Jul 7, 2009 (Bug fixes)
https://lists.linux-foundation.org/pipermail/containers/2009-July/019086.html
https://lists.linux-foundation.org/pipermail/containers/2009-July/019087.html
+ Patch from Gui Jianfeng on Jul 9, 2009 (per-cgroup requests limit)
https://lists.linux-foundation.org/pipermail/containers/2009-July/019123.html
+ Patch from me on Jul 16, 2009 (Bug fix)
https://lists.linux-foundation.org/pipermail/containers/2009-July/019286.html
+ 2 local bug fix patches
(Not posted yet, I'm posting them in following mails)
- All results are measured under nr_requests=500.
- Used fio to make I/O. Job file is like below. Used libaio and direct-I/O and tuned iodepth to make rl->count[1] approx 500 always.
----- fio job file : from here -----
[global]
size=128m
directory=/mnt/b1
runtime=30time_based
write_bw_log
bwavgtime=200
rw=randread
direct=1
ioengine=libaio
iodepth=500
[g1]
exec_prerun=./pre.sh /mnt/cgroups/g1
exec_postrun=./log.sh /mnt/cgroups/g1 sdb "_post"
[g2]
startdelay=10
exec_prerun=./pre.sh /mnt/cgroups/g2
exec_postrun=./log.sh /mnt/cgroups/g2 sdb "_post"
----- fio job file : till here -----
Note:pre.sh and log.sh used in exec_{pre|post}run are to assign processes to expected cgroups and record the conditions. Let me omit the detail of them because they are not fundamental part of this verification.
Results
====================
Bandwidth of g2 (=P2) was measured under some conditions. Conditions and bandwidth logs are shown below.
Bandwidth logs are shown only the beginning part (from starting of P2 to 3000[ms] after aprox.) because the full logs are so long. Average bandwidth from the beginning of log to ~10[sec] is also calculated.
Note1:
fio seems to log bandwidth only when actual data transfer occurs (correct me if it's not true). This means that there is no line with BW=0. In there is no data transfer, the time-stamp are simply skipped to record.
Note2:
Graph picture of the bandwidth logs is attached.
Result(1): orange
Result(2): green
Result(3): brown
Result(4): black
---------- Result (1) ----------
* Both of g1 and g2 have nr_group_requests=500
< Conditions >nr_requests = 500
g1/
io.nr_group_requests = 500
io.weight = 500
io.ioprio_class = 2
g2/
io.nr_group_requests = 500
io.weight = 500
io.ioprio_class = 2
< Bandwidth log of g2 >
t [ms] bw[KiB/s]
969 4
1170 1126
1374 1084
1576 876
1776 901
1980 1069
2191 1087
2400 1117
2612 1087
2822 1136
...
< Average bandwidth >
1063 [KiB/s]
(969~9979[ms])
---------- Result (2) ----------
* Both of g1 and g2 have nr_group_requests=100
< Conditions >nr_requests = 500
g1/
io.nr_group_requests = 100
io.weight = 500
io.ioprio_class = 2
g2/
io.nr_group_requests = 100
io.weight = 500
io.ioprio_class = 2
< Bandwidth log of g2 >
t [ms] bw[KiB/s]
1498 2
1733 892
2096 722
2311 1224
2534 1180
2753 1197
2988 1137
...
< Average bandwidth >
998 [KiB/s]
(1498~9898[ms])
---------- Result (3) ----------
* To set different nr_group_requests on g1 and g2
< Conditions >nr_requests = 500
g1/
io.nr_group_requests = 100
io.weight = 500
io.ioprio_class = 2
g2/
io.nr_group_requests = 500
io.weight = 500
io.ioprio_class = 2
< Bandwidth log of g2 >
t [ms] bw[KiB/s]
244 839
451 1133
659 964
877 1038
1088 1125
1294 979
1501 1068
1708 934
1916 1048
2117 1126
2328 1111
2533 1118
2758 1206
2969 990
...
< Average bandwidth >
1048 [KiB/s]
(244~9906[ms])
---------- Result (4) ----------
* To make g2/io.ioprio_class as RT
< Conditions >nr_requests = 500
g1/
io.nr_group_requests = 500
io.weight = 500
io.ioprio_class = 2
g2/
io.nr_group_requests = 500
io.weight = 500
io.ioprio_class = 1
< Bandwidth log of g2 >
t [ms] bw[KiB/s]
476 8
677 2211
878 2221
1080 2486
1281 2241
1481 2109
1681 2334
1882 2129
2082 2211
2283 1915
2489 1778
2690 1915
2891 1997
...
< Average bandwidth >
2132[KiB/s]
(476~9954[ms])
Consideration and Conclusion
=============================
From result(1), it is observed that it takes 1000~1200[ms] to rise
P2 bandwidth. In result(2), where both of g1 and g2 have nr_group_requests=100, the delay gets longer as 1800~2000[ms]. In addition to it, the average bandwidth becomes ~5% lower than result(1). This is supposed that P2 couldn't allocate enough requests. Then, result(3) shows that bandwidth of P2 can rise quickly (~300[ms]) if nr_group_requests can be set per-cgroup. Result(4) shows that the delay can be shortened by setting g2 as RT class, however, the delay is still longer than result(3).
I think it is confirmed that "per-cgroup nr_requests limitation is useful in a certain situation". Beyond that, the discussion topic is the benefit pointed out above is eligible for the complication of the implementation. IMHO, I don't think the implementation of per-cgroup request limitation is too complicated to accept. On the other hand I guess it suddenly gets complicated if we try to implement further more, especially hierarchical support. It is also true that I have a feeling that implementation without per-device limitation and hierarchical support is like "unfinished work".
So, my opinion so far is that, per-cgroup nr_requests limitation should be merged only if hierarchical support is concluded "unnecessary" for it. If merging it tempts hierarchical support, it shouldn't be. How about your opinion, all?
My considerations or verification method might be wrong. Please correct them if any. And if you have any other idea of scenario to verify the effect of per-cgroup nr_requests limitation, please let me know. I'll try it.
Signature · 3 lines
IKEDA, Munehiro
NEC Corporation of America
[email protected]Re: [PATCH] io-controller: implement per group request allocation limitation
Munehiro Ikeda wrote:
...
Quoted 24 lines
Consideration and Conclusion
=============================From result(1), it is observed that it takes 1000~1200[ms] to rise P2
bandwidth. In result(2), where both of g1 and g2 have nr_group_requests=100, the delay gets longer as 1800~2000[ms]. In addition to it, the average bandwidth becomes ~5% lower than result(1). This is supposed that P2 couldn't allocate enough requests. Then, result(3) shows that bandwidth of P2 can rise quickly (~300[ms]) ifnr_group_requestscan be set per-cgroup. Result(4) shows that the delay can be shortened by setting g2 as RT class, however, the delay is still longer than result(3).I think it is confirmed that "per-cgroup
nr_requestslimitation is useful in a certain situation". Beyond that, the discussion topic is the benefit pointed out above is eligible for the complication of the implementation. IMHO, I don't think the implementation of per-cgroup request limitation is too complicated to accept. On the other hand I guess it suddenly gets complicated if we try to implement further more, especially hierarchical support. It is also true that I have a feeling that implementation without per-device limitation and hierarchical support is like "unfinished work".
Quoted 4 lines
So, my opinion so far is that, per-cgroup
nr_requestslimitation should be merged only if hierarchical support is concluded "unnecessary" for it. If merging it tempts hierarchical support, it shouldn't be. How about your opinion, all?
Hi Munehiro-san,
Thanks for the great job. It seems Per-cgroup requests allocation limits has its value in some cases. IMHO, for the time being, we can just drop the hierarchical support for "Per-cgroup requests allocation limits", and see whether it can work well.
Quoted 11 lines
My considerations or verification method might be wrong. Please correct them if any. And if you have any other idea of scenario to verify the effect of per-cgroup
nr_requestslimitation, please let me know. I'll try it.------------------------------------------------------------------------
Signature · 2 lines
Regards Gui Jianfeng
A message that is not in the archive (sent off-list or not yet received): 4A77964A.7040602-MDRzhb/z0dd8UrSeD/[email protected]
Re: [PATCH] io-controller: implement per group request allocation limitation
Munehiro Ikeda wrote:
...
Quoted 24 lines
Consideration and Conclusion
=============================From result(1), it is observed that it takes 1000~1200[ms] to rise P2
bandwidth. In result(2), where both of g1 and g2 have nr_group_requests=100, the delay gets longer as 1800~2000[ms]. In addition to it, the average bandwidth becomes ~5% lower than result(1). This is supposed that P2 couldn't allocate enough requests. Then, result(3) shows that bandwidth of P2 can rise quickly (~300[ms]) ifnr_group_requestscan be set per-cgroup. Result(4) shows that the delay can be shortened by setting g2 as RT class, however, the delay is still longer than result(3).I think it is confirmed that "per-cgroup
nr_requestslimitation is useful in a certain situation". Beyond that, the discussion topic is the benefit pointed out above is eligible for the complication of the implementation. IMHO, I don't think the implementation of per-cgroup request limitation is too complicated to accept. On the other hand I guess it suddenly gets complicated if we try to implement further more, especially hierarchical support. It is also true that I have a feeling that implementation without per-device limitation and hierarchical support is like "unfinished work".
Quoted 4 lines
So, my opinion so far is that, per-cgroup
nr_requestslimitation should be merged only if hierarchical support is concluded "unnecessary" for it. If merging it tempts hierarchical support, it shouldn't be. How about your opinion, all?
Hi Munehiro-san,
Thanks for the great job. It seems Per-cgroup requests allocation limits has its value in some cases. IMHO, for the time being, we can just drop the hierarchical support for "Per-cgroup requests allocation limits", and see whether it can work well.
Quoted 11 lines
My considerations or verification method might be wrong. Please correct them if any. And if you have any other idea of scenario to verify the effect of per-cgroup
nr_requestslimitation, please let me know. I'll try it.------------------------------------------------------------------------
Signature · 2 lines
Regards Gui Jianfeng
Re: [PATCH] io-controller: implement per group request allocation limitation
In reply to a message not in the archiveQuoted 47 lines · Munehiro Ikeda
Gui Jianfeng wrote, on 07/14/2009 03:45 AM:
Quoted 46 lines
Quoted 14 lines · Munehiro Ikeda
Vivek Goyal wrote, on 07/13/2009 12:03 PM:
Quoted 7 lines · Gui Jianfeng
Quoted 6 lines
Hi Vivek,
This patch exports a cgroup based per group request limits interface. and removes the global one. Now we can use this interface to perform different request allocation limitation for different groups.
Quoted 6 lines
Thanks Gui. Few points come to mind.
- You seem to be making this as per cgroup limit on all devices. I guess that different devices in the system can have different settings of
q->nr_requestsand hence will probably want different per group limit. So we might have to make it per cgroup per device limit.Quoted 5 lines
From the viewpoint of implementation, there is a difficulty in my mind to
implement per cgroup per device limit arising from thatio_groupis allocated
when associated device is firstly used. I guess Gui chose per cgroup limit on all devices approach because of this, right?Yes, I choose this solution from the simplicity point of view, the code will get complicated if choosing per cgroup per device limit. But it seems per cgroup per device limits is more reasonable.
Quoted 20 lines
Quoted 10 lines
- There does not seem to be any checks for making sure that children cgroups don't have more request descriptors allocated than parent
group.- I am re-thinking that what's the advantage of configuring request descriptors also through cgroups. It does bring in additional
complexity
with it and it should justfiy the advantages. Can you think of some?Quoted 6 lines
Until and unless we can come up with some significant advantages, I
will
prefer to continue to use per group limit throughq->nr_group_requestsinterface instead of cgroup. Once things stablize, we can revisit
it and
see how this interface can be improved.I agree. I will try to clarify if per group per device limitation is needed
or not (or, if it has the advantage beyond the complexity) through some tests.
Quoted 6 lines
Quoted 1 line
Great, hope to hear you soon.
Sorry for so late. I tried it, and write the result and my opinion below...
Hi Ikeda,
Nice analysis. Few questions/comments inline...
Quoted 259 lines
Scenario
====================The possible scenario where per-cgroup
nr_requestslimitation is beneficial in my mind is that:- Process P1 in cgroup "g1" is running with submitting many requests to a device. The number of the requests in the device queue is almost
nr_requestsfor the device.- After a while, process P2 in cgroup "g2" starts running. P2 also tries to submit requests as many as P1.
- Assuming that user wants P2 to grab bandwidth as soon as possible and keep it certain level.
In this scenario, I predicted the bandwidth behavior of P2 along with tuning global
nr_group_requestslike below.- If having
nr_group_requestsalmost same asnr_requests, P1 can allocate requests up tonr_requestsand there is no room for P2 at the beginning of its running. As a result of it, P2 has to wait for a while till P1's requests are completed and rising of bandwidth is delayed.- If having
nr_group_requestsfewer to restrict requests from P1 and make room for P2, the bandwidth of P2 may be lower than the case that P1 can allocate more requests.If the prediction is correct and per-cgroup
nr_requestslimitation can make the situation better, per-cgroupnr_requestsis supposed to be beneficial.Verification Conditions
========================- Kernel:
2.6.31-rc1
+ Patches from Vivek on Jul 2, 2009
(IO scheduler based IO controller V6)https://lists.linux-foundation.org/pipermail/containers/2009-July/018948.html
+ Patches from Gui Jianfeng on Jul 7, 2009 (Bug fixes)https://lists.linux-foundation.org/pipermail/containers/2009-July/019086.html
https://lists.linux-foundation.org/pipermail/containers/2009-July/019087.html
+ Patch from Gui Jianfeng on Jul 9, 2009 (per-cgroup requests limit)https://lists.linux-foundation.org/pipermail/containers/2009-July/019123.html
+ Patch from me on Jul 16, 2009 (Bug fix)https://lists.linux-foundation.org/pipermail/containers/2009-July/019286.html
+ 2 local bug fix patches
(Not posted yet, I'm posting them in following mails)- All results are measured under nr_requests=500.
- Used fio to make I/O. Job file is like below. Used libaio and direct-I/O and tuned iodepth to make rl->count[1] approx 500 always.
----- fio job file : from here -----
[global]
size=128m
directory=/mnt/b1runtime=30
time_based
write_bw_log
bwavgtime=200rw=randread
direct=1
ioengine=libaio
iodepth=500[g1]
exec_prerun=./pre.sh/mnt/cgroups/g1
exec_postrun=./log.sh/mnt/cgroups/g1sdb "_post"[g2]
startdelay=10
exec_prerun=./pre.sh/mnt/cgroups/g2
exec_postrun=./log.sh/mnt/cgroups/g2sdb "_post"----- fio job file : till here -----
Note:
pre.shandlog.shused in exec_{pre|post}run are to assign processes to expected cgroups and record the conditions. Let me omit the detail of them because they are not fundamental part of this verification.Results
====================Bandwidth of g2 (=P2) was measured under some conditions. Conditions and bandwidth logs are shown below.
Bandwidth logs are shown only the beginning part (from starting of P2 to 3000[ms] after aprox.) because the full logs are so long. Average bandwidth from the beginning of log to ~10[sec] is also calculated.Note1:
fio seems to log bandwidth only when actual data transfer occurs (correct me if it's not true). This means that there is no line with BW=0. In there is no data transfer, the time-stamp are simply skipped to record.Note2:
Graph picture of the bandwidth logs is attached.
Result(1): orange
Result(2): green
Result(3): brown
Result(4): black---------- Result (1) ----------
* Both of g1 and g2 have nr_group_requests=500
< Conditions >
nr_requests= 500
g1/
io.nr_group_requests = 500
io.weight = 500
io.ioprio_class = 2
g2/
io.nr_group_requests = 500
io.weight = 500
io.ioprio_class = 2< Bandwidth log of g2 >
t [ms] bw[KiB/s]
969 4
1170 1126
1374 1084
1576 876
1776 901
1980 1069
2191 1087
2400 1117
2612 1087
2822 1136
...< Average bandwidth >
1063 [KiB/s]
(969~9979[ms])---------- Result (2) ----------
* Both of g1 and g2 have nr_group_requests=100
< Conditions >
nr_requests= 500
g1/
io.nr_group_requests = 100
io.weight = 500
io.ioprio_class = 2
g2/
io.nr_group_requests = 100
io.weight = 500
io.ioprio_class = 2< Bandwidth log of g2 >
t [ms] bw[KiB/s]
1498 2
1733 892
2096 722
2311 1224
2534 1180
2753 1197
2988 1137
...< Average bandwidth >
998 [KiB/s]
(1498~9898[ms])---------- Result (3) ----------
* To set different
nr_group_requestson g1 and g2< Conditions >
nr_requests= 500
g1/
io.nr_group_requests = 100
io.weight = 500
io.ioprio_class = 2
g2/
io.nr_group_requests = 500
io.weight = 500
io.ioprio_class = 2< Bandwidth log of g2 >
t [ms] bw[KiB/s]
244 839
451 1133
659 964
877 1038
1088 1125
1294 979
1501 1068
1708 934
1916 1048
2117 1126
2328 1111
2533 1118
2758 1206
2969 990
...< Average bandwidth >
1048 [KiB/s]
(244~9906[ms])---------- Result (4) ----------
* To make g2/io.ioprio_class as RT
< Conditions >
nr_requests= 500
g1/
io.nr_group_requests = 500
io.weight = 500
io.ioprio_class = 2
g2/
io.nr_group_requests = 500
io.weight = 500
io.ioprio_class = 1< Bandwidth log of g2 >
t [ms] bw[KiB/s]
476 8
677 2211
878 2221
1080 2486
1281 2241
1481 2109
1681 2334
1882 2129
2082 2211
2283 1915
2489 1778
2690 1915
2891 1997
...< Average bandwidth >
2132[KiB/s]
(476~9954[ms])
Quoted 6 lines
Consideration and Conclusion
=============================From result(1), it is observed that it takes 1000~1200[ms] to rise P2
bandwidth. In result(2), where both of g1 and g2 have nr_group_requests=100, the delay gets longer as 1800~2000[ms].
Result (2) is surprising in terms of delay. Queue limit is 500 and per group limit is 100. That means that group g2 should have got request descriptor allocated as soon as it sent a request to device. That should also mean that it should start getting serviced soon and BW available should rise soon.
I am not sure why reverse is happening. Sounds like a bug somewhere..
Quoted 10 lines
In
addition to it, the average bandwidth becomes ~5% lower than result(1). This is supposed that P2 couldn't allocate enough requests. Then, result(3) shows that bandwidth of P2 can rise quickly (~300[ms]) ifnr_group_requestscan be set per-cgroup. Result(4) shows that the delay can be shortened by setting g2 as RT class, however, the delay is still longer than result(3).I think it is confirmed that "per-cgroup
nr_requestslimitation is useful in a certain situation".
I am still not clear how per cgroup request descriptor limit is useful. Having higher number of descriptor reserved for a cgroup probably should help a bit on queuing hardware where more requests from same queue can be dispatched at one go. May be it can also help a bit in more merging in certain scenarios and gives higher throughput.
I think key question here is how relevant the number of request descriptors is when it comes to fairness for the group. I have not experimented but probably a few request descriptors (25-30) probably should be enough to ensure group gets fair amount of disk. That's a different thing that throughput of group might suffer and bandwith allocation among task with-in group will also have adverse effect.
Quoted 9 lines
Beyond that, the discussion topic is
the benefit pointed out above is eligible for the complication of the implementation. IMHO, I don't think the implementation of per-cgroup request limitation is too complicated to accept. On the other hand I guess it suddenly gets complicated if we try to implement further more, especially hierarchical support. It is also true that I have a feeling that implementation without per-device limitation and hierarchical support is like "unfinished work".
IMHO, configuring request descriptors per cgroup can be a future TODO item depending on how useful people consider it. For the time being, it would be better to keep things simple and code small.
In fact currently I am trying to replace BFQ with a CFS style scheduler in io controller patches to reduce the code size as well as its complexity.
Thanks
Vivek
Quoted 18 lines
So, my opinion so far is that, per-cgroup
nr_requestslimitation should be merged only if hierarchical support is concluded "unnecessary" for it.
If merging it tempts hierarchical support, it shouldn't be.
How about your opinion, all?My considerations or verification method might be wrong. Please correct them if any. And if you have any other idea of scenario to verify the effect of per-cgroup
nr_requestslimitation, please let me know. I'll try it.Signature · 3 lines
IKEDA, Munehiro NEC Corporation of America m-ikeda-MDRzhb/z0dd8UrSeD/[email protected]
Re: [PATCH] io-controller: implement per group request allocation limitation
In reply to Munehiro IkedaQuoted 47 lines · Munehiro Ikeda
Gui Jianfeng wrote, on 07/14/2009 03:45 AM:
Quoted 46 lines
Quoted 14 lines · Munehiro Ikeda
Vivek Goyal wrote, on 07/13/2009 12:03 PM:
Quoted 7 lines · Gui Jianfeng
Quoted 6 lines
Hi Vivek,
This patch exports a cgroup based per group request limits interface. and removes the global one. Now we can use this interface to perform different request allocation limitation for different groups.
Quoted 6 lines
Thanks Gui. Few points come to mind.
- You seem to be making this as per cgroup limit on all devices. I guess that different devices in the system can have different settings of
q->nr_requestsand hence will probably want different per group limit. So we might have to make it per cgroup per device limit.Quoted 5 lines
From the viewpoint of implementation, there is a difficulty in my mind to
implement per cgroup per device limit arising from thatio_groupis allocated
when associated device is firstly used. I guess Gui chose per cgroup limit on all devices approach because of this, right?Yes, I choose this solution from the simplicity point of view, the code will get complicated if choosing per cgroup per device limit. But it seems per cgroup per device limits is more reasonable.
Quoted 20 lines
Quoted 10 lines
- There does not seem to be any checks for making sure that children cgroups don't have more request descriptors allocated than parent
group.- I am re-thinking that what's the advantage of configuring request descriptors also through cgroups. It does bring in additional
complexity
with it and it should justfiy the advantages. Can you think of some?Quoted 6 lines
Until and unless we can come up with some significant advantages, I
will
prefer to continue to use per group limit throughq->nr_group_requestsinterface instead of cgroup. Once things stablize, we can revisit
it and
see how this interface can be improved.I agree. I will try to clarify if per group per device limitation is needed
or not (or, if it has the advantage beyond the complexity) through some tests.
Quoted 6 lines
Quoted 1 line
Great, hope to hear you soon.
Sorry for so late. I tried it, and write the result and my opinion below...
Hi Ikeda,
Nice analysis. Few questions/comments inline...
Quoted 259 lines
Scenario
====================The possible scenario where per-cgroup
nr_requestslimitation is beneficial in my mind is that:- Process P1 in cgroup "g1" is running with submitting many requests to a device. The number of the requests in the device queue is almost
nr_requestsfor the device.- After a while, process P2 in cgroup "g2" starts running. P2 also tries to submit requests as many as P1.
- Assuming that user wants P2 to grab bandwidth as soon as possible and keep it certain level.
In this scenario, I predicted the bandwidth behavior of P2 along with tuning global
nr_group_requestslike below.- If having
nr_group_requestsalmost same asnr_requests, P1 can allocate requests up tonr_requestsand there is no room for P2 at the beginning of its running. As a result of it, P2 has to wait for a while till P1's requests are completed and rising of bandwidth is delayed.- If having
nr_group_requestsfewer to restrict requests from P1 and make room for P2, the bandwidth of P2 may be lower than the case that P1 can allocate more requests.If the prediction is correct and per-cgroup
nr_requestslimitation can make the situation better, per-cgroupnr_requestsis supposed to be beneficial.Verification Conditions
========================- Kernel:
2.6.31-rc1
+ Patches from Vivek on Jul 2, 2009
(IO scheduler based IO controller V6)https://lists.linux-foundation.org/pipermail/containers/2009-July/018948.html
+ Patches from Gui Jianfeng on Jul 7, 2009 (Bug fixes)https://lists.linux-foundation.org/pipermail/containers/2009-July/019086.html
https://lists.linux-foundation.org/pipermail/containers/2009-July/019087.html
+ Patch from Gui Jianfeng on Jul 9, 2009 (per-cgroup requests limit)https://lists.linux-foundation.org/pipermail/containers/2009-July/019123.html
+ Patch from me on Jul 16, 2009 (Bug fix)https://lists.linux-foundation.org/pipermail/containers/2009-July/019286.html
+ 2 local bug fix patches
(Not posted yet, I'm posting them in following mails)- All results are measured under nr_requests=500.
- Used fio to make I/O. Job file is like below. Used libaio and direct-I/O and tuned iodepth to make rl->count[1] approx 500 always.
----- fio job file : from here -----
[global]
size=128m
directory=/mnt/b1runtime=30
time_based
write_bw_log
bwavgtime=200rw=randread
direct=1
ioengine=libaio
iodepth=500[g1]
exec_prerun=./pre.sh/mnt/cgroups/g1
exec_postrun=./log.sh/mnt/cgroups/g1sdb "_post"[g2]
startdelay=10
exec_prerun=./pre.sh/mnt/cgroups/g2
exec_postrun=./log.sh/mnt/cgroups/g2sdb "_post"----- fio job file : till here -----
Note:
pre.shandlog.shused in exec_{pre|post}run are to assign processes to expected cgroups and record the conditions. Let me omit the detail of them because they are not fundamental part of this verification.Results
====================Bandwidth of g2 (=P2) was measured under some conditions. Conditions and bandwidth logs are shown below.
Bandwidth logs are shown only the beginning part (from starting of P2 to 3000[ms] after aprox.) because the full logs are so long. Average bandwidth from the beginning of log to ~10[sec] is also calculated.Note1:
fio seems to log bandwidth only when actual data transfer occurs (correct me if it's not true). This means that there is no line with BW=0. In there is no data transfer, the time-stamp are simply skipped to record.Note2:
Graph picture of the bandwidth logs is attached.
Result(1): orange
Result(2): green
Result(3): brown
Result(4): black---------- Result (1) ----------
* Both of g1 and g2 have nr_group_requests=500
< Conditions >
nr_requests= 500
g1/
io.nr_group_requests = 500
io.weight = 500
io.ioprio_class = 2
g2/
io.nr_group_requests = 500
io.weight = 500
io.ioprio_class = 2< Bandwidth log of g2 >
t [ms] bw[KiB/s]
969 4
1170 1126
1374 1084
1576 876
1776 901
1980 1069
2191 1087
2400 1117
2612 1087
2822 1136
...< Average bandwidth >
1063 [KiB/s]
(969~9979[ms])---------- Result (2) ----------
* Both of g1 and g2 have nr_group_requests=100
< Conditions >
nr_requests= 500
g1/
io.nr_group_requests = 100
io.weight = 500
io.ioprio_class = 2
g2/
io.nr_group_requests = 100
io.weight = 500
io.ioprio_class = 2< Bandwidth log of g2 >
t [ms] bw[KiB/s]
1498 2
1733 892
2096 722
2311 1224
2534 1180
2753 1197
2988 1137
...< Average bandwidth >
998 [KiB/s]
(1498~9898[ms])---------- Result (3) ----------
* To set different
nr_group_requestson g1 and g2< Conditions >
nr_requests= 500
g1/
io.nr_group_requests = 100
io.weight = 500
io.ioprio_class = 2
g2/
io.nr_group_requests = 500
io.weight = 500
io.ioprio_class = 2< Bandwidth log of g2 >
t [ms] bw[KiB/s]
244 839
451 1133
659 964
877 1038
1088 1125
1294 979
1501 1068
1708 934
1916 1048
2117 1126
2328 1111
2533 1118
2758 1206
2969 990
...< Average bandwidth >
1048 [KiB/s]
(244~9906[ms])---------- Result (4) ----------
* To make g2/io.ioprio_class as RT
< Conditions >
nr_requests= 500
g1/
io.nr_group_requests = 500
io.weight = 500
io.ioprio_class = 2
g2/
io.nr_group_requests = 500
io.weight = 500
io.ioprio_class = 1< Bandwidth log of g2 >
t [ms] bw[KiB/s]
476 8
677 2211
878 2221
1080 2486
1281 2241
1481 2109
1681 2334
1882 2129
2082 2211
2283 1915
2489 1778
2690 1915
2891 1997
...< Average bandwidth >
2132[KiB/s]
(476~9954[ms])
Quoted 6 lines
Consideration and Conclusion
=============================From result(1), it is observed that it takes 1000~1200[ms] to rise P2
bandwidth. In result(2), where both of g1 and g2 have nr_group_requests=100, the delay gets longer as 1800~2000[ms].
Result (2) is surprising in terms of delay. Queue limit is 500 and per group limit is 100. That means that group g2 should have got request descriptor allocated as soon as it sent a request to device. That should also mean that it should start getting serviced soon and BW available should rise soon.
I am not sure why reverse is happening. Sounds like a bug somewhere..
Quoted 10 lines
In
addition to it, the average bandwidth becomes ~5% lower than result(1). This is supposed that P2 couldn't allocate enough requests. Then, result(3) shows that bandwidth of P2 can rise quickly (~300[ms]) ifnr_group_requestscan be set per-cgroup. Result(4) shows that the delay can be shortened by setting g2 as RT class, however, the delay is still longer than result(3).I think it is confirmed that "per-cgroup
nr_requestslimitation is useful in a certain situation".
I am still not clear how per cgroup request descriptor limit is useful. Having higher number of descriptor reserved for a cgroup probably should help a bit on queuing hardware where more requests from same queue can be dispatched at one go. May be it can also help a bit in more merging in certain scenarios and gives higher throughput.
I think key question here is how relevant the number of request descriptors is when it comes to fairness for the group. I have not experimented but probably a few request descriptors (25-30) probably should be enough to ensure group gets fair amount of disk. That's a different thing that throughput of group might suffer and bandwith allocation among task with-in group will also have adverse effect.
Quoted 9 lines
Beyond that, the discussion topic is
the benefit pointed out above is eligible for the complication of the implementation. IMHO, I don't think the implementation of per-cgroup request limitation is too complicated to accept. On the other hand I guess it suddenly gets complicated if we try to implement further more, especially hierarchical support. It is also true that I have a feeling that implementation without per-device limitation and hierarchical support is like "unfinished work".
IMHO, configuring request descriptors per cgroup can be a future TODO item depending on how useful people consider it. For the time being, it would be better to keep things simple and code small.
In fact currently I am trying to replace BFQ with a CFS style scheduler in io controller patches to reduce the code size as well as its complexity.
Thanks
Vivek
Quoted 18 lines
So, my opinion so far is that, per-cgroup
nr_requestslimitation should be merged only if hierarchical support is concluded "unnecessary" for it.
If merging it tempts hierarchical support, it shouldn't be.
How about your opinion, all?My considerations or verification method might be wrong. Please correct them if any. And if you have any other idea of scenario to verify the effect of per-cgroup
nr_requestslimitation, please let me know. I'll try it.Signature · 3 lines
IKEDA, Munehiro NEC Corporation of America [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: [PATCH] io-controller: implement per group request allocation limitation
Gui Jianfeng wrote, on 07/14/2009 03:45 AM:
Quoted 41 lines
Quoted 14 lines · Munehiro Ikeda
Vivek Goyal wrote, on 07/13/2009 12:03 PM:
Quoted 7 lines · Gui Jianfeng
Quoted 6 lines
Hi Vivek,
This patch exports a cgroup based per group request limits interface. and removes the global one. Now we can use this interface to perform different request allocation limitation for different groups.
Quoted 6 lines
Thanks Gui. Few points come to mind.
- You seem to be making this as per cgroup limit on all devices. I guess that different devices in the system can have different settings of
q->nr_requestsand hence will probably want different per group limit. So we might have to make it per cgroup per device limit.Quoted 5 lines
From the viewpoint of implementation, there is a difficulty in my mind to
implement per cgroup per device limit arising from thatio_groupis allocated
when associated device is firstly used. I guess Gui chose per cgroup limit on all devices approach because of this, right?Yes, I choose this solution from the simplicity point of view, the code will get complicated if choosing per cgroup per device limit. But it seems per cgroup per device limits is more reasonable.
Quoted 16 lines
Quoted 16 lines
- There does not seem to be any checks for making sure that children cgroups don't have more request descriptors allocated than parent
group.- I am re-thinking that what's the advantage of configuring request descriptors also through cgroups. It does bring in additional
complexity
with it and it should justfiy the advantages. Can you think of some?Until and unless we can come up with some significant advantages, I
will
prefer to continue to use per group limit throughq->nr_group_requestsinterface instead of cgroup. Once things stablize, we can revisit
it and
see how this interface can be improved.
Quoted 6 lines
Quoted 4 lines
I agree. I will try to clarify if per group per device limitation is needed
or not (or, if it has the advantage beyond the complexity) through some tests.Great, hope to hear you soon.
Sorry for so late. I tried it, and write the result and my opinion below...
Scenario
====================
The possible scenario where per-cgroup nr_requests limitation is beneficial in my mind is that:
- Process P1 in cgroup "g1" is running with submitting many requests to a device. The number of the requests in the device queue is almost nr_requests for the device.
- After a while, process P2 in cgroup "g2" starts running. P2 also tries to submit requests as many as P1.
- Assuming that user wants P2 to grab bandwidth as soon as possible and keep it certain level.
In this scenario, I predicted the bandwidth behavior of P2 along with tuning global nr_group_requests like below.
- If having nr_group_requests almost same as nr_requests, P1 can allocate requests up to nr_requests and there is no room for P2 at the beginning of its running. As a result of it, P2 has to wait for a while till P1's requests are completed and rising of bandwidth is delayed.
- If having nr_group_requests fewer to restrict requests from P1 and make room for P2, the bandwidth of P2 may be lower than the case that P1 can allocate more requests.
If the prediction is correct and per-cgroup nr_requests limitation can make the situation better, per-cgroup nr_requests is supposed to be beneficial.
Verification Conditions
========================
- Kernel:
2.6.31-rc1
+ Patches from Vivek on Jul 2, 2009
(IO scheduler based IO controller V6)
https://lists.linux-foundation.org/pipermail/containers/2009-July/018948.html
+ Patches from Gui Jianfeng on Jul 7, 2009 (Bug fixes)
https://lists.linux-foundation.org/pipermail/containers/2009-July/019086.html
https://lists.linux-foundation.org/pipermail/containers/2009-July/019087.html
+ Patch from Gui Jianfeng on Jul 9, 2009 (per-cgroup requests limit)
https://lists.linux-foundation.org/pipermail/containers/2009-July/019123.html
+ Patch from me on Jul 16, 2009 (Bug fix)
https://lists.linux-foundation.org/pipermail/containers/2009-July/019286.html
+ 2 local bug fix patches
(Not posted yet, I'm posting them in following mails)
- All results are measured under nr_requests=500.
- Used fio to make I/O. Job file is like below. Used libaio and direct-I/O and tuned iodepth to make rl->count[1] approx 500 always.
----- fio job file : from here -----
[global]
size=128m
directory=/mnt/b1
runtime=30time_based
write_bw_log
bwavgtime=200
rw=randread
direct=1
ioengine=libaio
iodepth=500
[g1]
exec_prerun=./pre.sh /mnt/cgroups/g1
exec_postrun=./log.sh /mnt/cgroups/g1 sdb "_post"
[g2]
startdelay=10
exec_prerun=./pre.sh /mnt/cgroups/g2
exec_postrun=./log.sh /mnt/cgroups/g2 sdb "_post"
----- fio job file : till here -----
Note:pre.sh and log.sh used in exec_{pre|post}run are to assign processes to expected cgroups and record the conditions. Let me omit the detail of them because they are not fundamental part of this verification.
Results
====================
Bandwidth of g2 (=P2) was measured under some conditions. Conditions and bandwidth logs are shown below.
Bandwidth logs are shown only the beginning part (from starting of P2 to 3000[ms] after aprox.) because the full logs are so long. Average bandwidth from the beginning of log to ~10[sec] is also calculated.
Note1:
fio seems to log bandwidth only when actual data transfer occurs (correct me if it's not true). This means that there is no line with BW=0. In there is no data transfer, the time-stamp are simply skipped to record.
Note2:
Graph picture of the bandwidth logs is attached.
Result(1): orange
Result(2): green
Result(3): brown
Result(4): black
---------- Result (1) ----------
* Both of g1 and g2 have nr_group_requests=500
< Conditions >nr_requests = 500
g1/
io.nr_group_requests = 500
io.weight = 500
io.ioprio_class = 2
g2/
io.nr_group_requests = 500
io.weight = 500
io.ioprio_class = 2
< Bandwidth log of g2 >
t [ms] bw[KiB/s]
969 4
1170 1126
1374 1084
1576 876
1776 901
1980 1069
2191 1087
2400 1117
2612 1087
2822 1136
...
< Average bandwidth >
1063 [KiB/s]
(969~9979[ms])
---------- Result (2) ----------
* Both of g1 and g2 have nr_group_requests=100
< Conditions >nr_requests = 500
g1/
io.nr_group_requests = 100
io.weight = 500
io.ioprio_class = 2
g2/
io.nr_group_requests = 100
io.weight = 500
io.ioprio_class = 2
< Bandwidth log of g2 >
t [ms] bw[KiB/s]
1498 2
1733 892
2096 722
2311 1224
2534 1180
2753 1197
2988 1137
...
< Average bandwidth >
998 [KiB/s]
(1498~9898[ms])
---------- Result (3) ----------
* To set different nr_group_requests on g1 and g2
< Conditions >nr_requests = 500
g1/
io.nr_group_requests = 100
io.weight = 500
io.ioprio_class = 2
g2/
io.nr_group_requests = 500
io.weight = 500
io.ioprio_class = 2
< Bandwidth log of g2 >
t [ms] bw[KiB/s]
244 839
451 1133
659 964
877 1038
1088 1125
1294 979
1501 1068
1708 934
1916 1048
2117 1126
2328 1111
2533 1118
2758 1206
2969 990
...
< Average bandwidth >
1048 [KiB/s]
(244~9906[ms])
---------- Result (4) ----------
* To make g2/io.ioprio_class as RT
< Conditions >nr_requests = 500
g1/
io.nr_group_requests = 500
io.weight = 500
io.ioprio_class = 2
g2/
io.nr_group_requests = 500
io.weight = 500
io.ioprio_class = 1
< Bandwidth log of g2 >
t [ms] bw[KiB/s]
476 8
677 2211
878 2221
1080 2486
1281 2241
1481 2109
1681 2334
1882 2129
2082 2211
2283 1915
2489 1778
2690 1915
2891 1997
...
< Average bandwidth >
2132[KiB/s]
(476~9954[ms])
Consideration and Conclusion
=============================
From result(1), it is observed that it takes 1000~1200[ms] to rise
P2 bandwidth. In result(2), where both of g1 and g2 have nr_group_requests=100, the delay gets longer as 1800~2000[ms]. In addition to it, the average bandwidth becomes ~5% lower than result(1). This is supposed that P2 couldn't allocate enough requests. Then, result(3) shows that bandwidth of P2 can rise quickly (~300[ms]) if nr_group_requests can be set per-cgroup. Result(4) shows that the delay can be shortened by setting g2 as RT class, however, the delay is still longer than result(3).
I think it is confirmed that "per-cgroup nr_requests limitation is useful in a certain situation". Beyond that, the discussion topic is the benefit pointed out above is eligible for the complication of the implementation. IMHO, I don't think the implementation of per-cgroup request limitation is too complicated to accept. On the other hand I guess it suddenly gets complicated if we try to implement further more, especially hierarchical support. It is also true that I have a feeling that implementation without per-device limitation and hierarchical support is like "unfinished work".
So, my opinion so far is that, per-cgroup nr_requests limitation should be merged only if hierarchical support is concluded "unnecessary" for it. If merging it tempts hierarchical support, it shouldn't be. How about your opinion, all?
My considerations or verification method might be wrong. Please correct them if any. And if you have any other idea of scenario to verify the effect of per-cgroup nr_requests limitation, please let me know. I'll try it.
Signature · 3 lines
IKEDA, Munehiro
NEC Corporation of America
m-ikeda-MDRzhb/z0dd8UrSeD/[email protected]_______________________________________________
Containers mailing list Containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org
https://lists.linux-foundation.org/mailman/listinfo/containers
A message that is not in the archive (sent off-list or not yet received): 4A5BA238.3030902-MDRzhb/z0dd8UrSeD/[email protected]
Re: [PATCH] io-controller: implement per group request allocation limitation
Quoted 16 lines · Munehiro Ikeda
Vivek Goyal wrote, on 07/13/2009 12:03 PM:
Quoted 8 lines · Gui Jianfeng
Quoted 6 lines
Hi Vivek,
This patch exports a cgroup based per group request limits interface. and removes the global one. Now we can use this interface to perform different request allocation limitation for different groups.
Quoted 6 lines
Thanks Gui. Few points come to mind.
- You seem to be making this as per cgroup limit on all devices. I guess that different devices in the system can have different settings of
q->nr_requestsand hence will probably want different per group limit. So we might have to make it per cgroup per device limit.
Quoted 5 lines
From the viewpoint of implementation, there is a difficulty in my mind to implement per cgroup per device limit arising from that
io_groupis allocated
when associated device is firstly used. I guess Gui chose per cgroup limit on all devices approach because of this, right?
Yes, I choose this solution from the simplicity point of view, the code will get complicated if choosing per cgroup per device limit. But it seems per cgroup per device limits is more reasonable.
Quoted 18 lines
Quoted 11 lines
- There does not seem to be any checks for making sure that children cgroups don't have more request descriptors allocated than parent
group.- I am re-thinking that what's the advantage of configuring request descriptors also through cgroups. It does bring in additional
complexity
with it and it should justfiy the advantages. Can you think of some?Quoted 6 lines
Until and unless we can come up with some significant advantages, I
will
prefer to continue to use per group limit throughq->nr_group_requestsinterface instead of cgroup. Once things stablize, we can revisit
it and
see how this interface can be improved.
Quoted 4 lines
I agree. I will try to clarify if per group per device limitation is needed
or not (or, if it has the advantage beyond the complexity) through some tests.
Great, hope to hear you soon.
Signature · 2 lines
Regards Gui Jianfeng
Quoted 368 lines
Tnaks a lot,
MuuhhQuoted 360 lines
Thanks
VivekQuoted 356 lines
Signed-off-by Gui Jianfeng<[email protected]>--- block/blk-core.c | 23 ++++++++++-- block/blk-settings.c | 1 - block/blk-sysfs.c | 43 ----------------------- block/elevator-fq.c | 94 ++++++++++++++++++++++++++++++++++++++++++++++--- block/elevator-fq.h | 4 ++ 5 files changed, 111 insertions(+), 54 deletions(-)block/blk-core.c +8
-1@@ -722,13 +722,20 @@ static void ioc_set_batching(struct request_queue *q, struct io_context *ioc) static void __freed_request(struct request_queue *q, int sync, struct request_list *rl) { + struct io_group *iog; +unsigned long nr_group_requests; +if (q->rq_data.count[sync]<queue_congestion_off_threshold (q))blk_clear_queue_congested (q, sync);if (q->rq_data.count[sync] +1 <= q->nr_requests)blk_clear_queue_full (q, sync); - if (rl->count[sync] + 1<= q->nr_group_requests) { + iog =rl_iog (rl); + + nr_group_requests =get_group_requests (q, iog); + +if (nr_group_requests&& rl->count[sync] +1 <= nr_group_requests) {if (waitqueue_active (&rl->wait[sync]))wake_up (&rl->wait[sync]);}block/blk-core.c +2
-0@@ -828,6 +835,8 @@ static struct request *get_request(struct request_queue *q, int rw_flags, const bool is_sync = rw_is_sync(rw_flags) != 0;int may_queue, priv;int sleep_on_global =0 ; +struct io_group *iog; +unsigned long nr_group_requests; may_queue =elv_may_queue (q, rw_flags);if (may_queue == ELV_MQUEUE_NO)block/blk-core.c +6
-1@@ -843,7 +852,12 @@ static struct request *get_request(struct request_queue *q, int rw_flags, if (q->rq_data.count[is_sync]+1>= q->nr_requests) blk_set_queue_full(q, is_sync); -if (rl->count[is_sync]+1 >= q->nr_group_requests) { + iog = rl_iog(rl); + + nr_group_requests =get_group_requests (q, iog); + +if (nr_group_requests&& + rl->count[is_sync]+1 >= nr_group_requests) { ioc =current_io_context (GFP_ATOMIC , q->node);/* * The queue request descriptor group will fill after this
block/blk-core.c +1
-1@@ -852,7 +866,7 @@ static struct request *get_request(structrequest_queue *q, int rw_flags, * This process will be allowed to complete a batch of * requests, others will be blocked. */ -if (rl->count[is_sync]<= q->nr_group_requests) +if (rl->count[is_sync]<= nr_group_requests)ioc_set_batching (q, ioc);else {if (may_queue != ELV_MQUEUE_MUSTblock/blk-core.c +2
-1@@ -898,7 +912,8 @@ static struct request *get_request(structrequest_queue *q, int rw_flags, * from per group request list */ -if (rl->count[is_sync]>= (3 * q->nr_group_requests /2 )) +if (nr_group_requests&& + rl->count[is_sync]>= (3 * nr_group_requests /2 )) goto out;rl->starved[is_sync] = 0;block/blk-settings.c +0
-1@@ -148,7 +148,6 @@ void blk_queue_make_request(struct request_queue*q, make_request_fn *mfn) * set defaults */ q->nr_requests =BLKDEV_MAX_RQ ; - q->nr_group_requests =BLKDEV_MAX_GROUP_RQ ; q->make_request_fn = mfn;blk_queue_dma_alignment(q, 511);block/blk-sysfs.c +0
-32@@ -78,40 +78,8 @@ queue_requests_store(struct request_queue *q, const char *page, size_t count) return ret; }#ifdef CONFIG_GROUP_IOSCHED -static ssize_t queue_group_requests_show (struct request_queue *q, char *page) -{ -return queue_var_show (q->nr_group_requests, (page)); -} -extern void elv_io_group_congestion_threshold (struct request_queue *q,struct io_group *iog); - -static ssize_t -queue_group_requests_store (struct request_queue *q,const char *page, -size_t count) -{ -struct hlist_node *n; -struct io_group *iog; -struct elv_fq_data *efqd; -unsigned long nr; -int ret =queue_var_store (&nr, page, count); - -if (nr<BLKDEV_MIN_RQ ) - nr =BLKDEV_MIN_RQ ; - -spin_lock_irq (q->queue_lock); - - q->nr_group_requests = nr; - - efqd =&q->elevator->efqd; - -hlist_for_each_entry (iog, n,&efqd->group_list, elv_data_node) { -elv_io_group_congestion_threshold (q, iog); - } - -spin_unlock_irq (q->queue_lock); -return ret; -}#endif static
ssize_tqueue_ra_show(struct request_queue *q, char *page)block/blk-sysfs.c +0
-8@@ -278,14 +246,6 @@ static struct queue_sysfs_entry queue_requests_entry = { .store = queue_requests_store, }; -#ifdef CONFIG_GROUP_IOSCHED -static struct queue_sysfs_entry queue_group_requests_entry = { - .attr = {.name ="nr_group_requests" , .mode =S_IRUGO |S_IWUSR }, - .show = queue_group_requests_show, - .store = queue_group_requests_store, -}; -#endif -static struct queue_sysfs_entry queue_ra_entry = { .attr = {.name ="read_ahead_kb" , .mode =S_IRUGO |S_IWUSR },.show =
queue_ra_show,block/blk-sysfs.c +0
-3@@ -360,9 +320,6 @@ static struct queue_sysfs_entry queue_iostats_entry = {static struct attribute *default_attrs[] = { &queue_requests_entry.attr, -#ifdef CONFIG_GROUP_IOSCHED - &queue_group_requests_entry.attr, -#endif &queue_ra_entry.attr, &queue_max_hw_sectors_entry.attr,&queue_max_sectors_entry.attr,
block/elevator-fq.c +29
-0@@ -59,6 +59,35 @@ elv_release_ioq(struct elevator_queue *eq, struct io_queue **ioq_ptr);#define for_each_entity_safe (entity, parent) \ for (; entity&& ({ parent = entity->parent; 1; }); entity = parent) +unsigned short get_group_requests (struct request_queue *q, +struct io_group *iog) +{ +struct cgroup_subsys_state *css; +struct io_cgroup *iocg; +unsigned long nr_group_requests; + +if (!iog) +return q->nr_requests; + +rcu_read_lock (); + +if (!iog->iocg_id) { + nr_group_requests =0 ; + goto out; + } + + css =css_lookup (&io_subsys, iog->iocg_id); +if (!css) { + nr_group_requests =0 ; + goto out; + } + + iocg = container_of(css, struct io_cgroup, css); + nr_group_requests = iocg->nr_group_requests; +out: +rcu_read_unlock (); +return nr_group_requests; +}static struct
io_entity*bfq_lookup_next_entity(structio_sched_data*sd,int extract);block/elevator-fq.c +8
-5@@ -1257,14 +1286,17 @@ void elv_io_group_congestion_threshold(structrequest_queue *q, struct io_group *iog) { int nr; +unsigned long nr_group_requests; - nr = q->nr_group_requests - (q->nr_group_requests /8 ) +1 ; -if (nr> q->nr_group_requests) - nr = q->nr_group_requests; + nr_group_requests =get_group_requests (q, iog); + + nr = nr_group_requests - (nr_group_requests /8 ) +1 ; +if (nr> nr_group_requests) + nr = nr_group_requests; iog->nr_congestion_on = nr; - nr = q->nr_group_requests - (q->nr_group_requests /8 ) - - (q->nr_group_requests /16 ) -1 ; + nr = nr_group_requests - (nr_group_requests /8 ) + - (nr_group_requests /16 ) -1 ;if (nr<1 ) nr =1 ;iog->nr_congestion_off = nr;block/elevator-fq.c +2
-0@@ -1283,6 +1315,7 @@ int elv_io_group_congested(struct request_queue *q, struct page *page, int sync) {struct io_group *iog;int ret =0 ; +unsigned long nr_group_requests;rcu_read_lock ();@@ -1300,10 +1333,11 @@ int elv_io_group_congested(struct request_queue *q, struct page *page, int sync) } ret =elv_is_iog_congested (q, iog, sync); + nr_group_requests =get_group_requests (q, iog);if (ret) elv_log_iog(&q->elevator->efqd, iog, "iog congested=%dsync=%d"
" rl.count[sync]=%d nr_group_requests=%d",block/elevator-fq.c +48
-1- ret, sync, iog->rl.count[sync], q->nr_group_requests); + ret, sync, iog->rl.count[sync], nr_group_requests);rcu_read_unlock ();return ret; }@@ -1549,6 +1583,48 @@ free_buf:return ret; } +static u64 io_cgroup_nr_requests_read (struct cgroup *cgroup, +struct cftype *cftype) +{ +struct io_cgroup *iocg; +u64 ret; + +if (!cgroup_lock_live_group (cgroup)) +return -ENODEV ; + + iocg =cgroup_to_io_cgroup (cgroup); +spin_lock_irq (&iocg->lock); + ret = iocg->nr_group_requests; +spin_unlock_irq (&iocg->lock); + +cgroup_unlock (); + +return ret; +} + +static int io_cgroup_nr_requests_write (struct cgroup *cgroup, +struct cftype *cftype, +u64 val) +{ +struct io_cgroup *iocg; + +if (val<BLKDEV_MIN_RQ ) + val =BLKDEV_MIN_RQ ; + +if (!cgroup_lock_live_group (cgroup)) +return -ENODEV ; + + iocg =cgroup_to_io_cgroup (cgroup); + +spin_lock_irq (&iocg->lock); + iocg->nr_group_requests = (unsigned long )val; +spin_unlock_irq (&iocg->lock); + +cgroup_unlock (); + +return 0 ; +} +#define SHOW_FUNCTION (__VAR) \ static u64 io_cgroup_##__VAR##_read(struct cgroup *cgroup, \ struct cftype *cftype) \@@ -1735,6 +1811,11 @@ static int io_cgroup_disk_dequeue_read(structcgroup *cgroup, struct cftype bfqio_files[] = { { + .name ="nr_group_requests" , + .read_u64 = io_cgroup_nr_requests_read, + .write_u64 = io_cgroup_nr_requests_write, + }, + { .name ="policy" , .read_seq_string = io_cgroup_policy_read,.write_string =
io_cgroup_policy_write,block/elevator-fq.c +1
-0@@ -1790,6 +1871,7 @@ static struct cgroup_subsys_state *iocg_create(struct cgroup_subsys *subsys, spin_lock_init(&iocg->lock);INIT_HLIST_HEAD (&iocg->group_data); + iocg->nr_group_requests =BLKDEV_MAX_GROUP_RQ ; iocg->weight =IO_DEFAULT_GRP_WEIGHT ; iocg->ioprio_class =IO_DEFAULT_GRP_CLASS ;INIT_LIST_HEAD(&iocg->policy_list);block/elevator-fq.h +4
-0@@ -308,6 +308,7 @@ struct io_cgroup {unsigned int weight;unsigned short ioprio_class; +unsigned long nr_group_requests;/* list of io_policy_node */ struct list_head policy_list;@@ -386,6 +387,9 @@ struct elv_fq_data {unsigned int fairness; }; +extern unsigned short get_group_requests (struct request_queue *q, +struct io_group *iog); +/* Logging facilities. */ #ifdef CONFIG_DEBUG_GROUP_IOSCHED #define elv_log_ioq(efqd, ioq, fmt, args...) \Signature · 1 line
1.5.4.rc3
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: [PATCH] io-controller: implement per group request allocation limitation
Vivek Goyal wrote, on 07/13/2009 12:03 PM:
Quoted 8 lines · Gui Jianfeng
Quoted 6 lines
Hi Vivek,
This patch exports a cgroup based per group request limits interface. and removes the global one. Now we can use this interface to perform different request allocation limitation for different groups.
Quoted 6 lines
Thanks Gui. Few points come to mind.
- You seem to be making this as per cgroup limit on all devices. I guess that different devices in the system can have different settings of
q->nr_requestsand hence will probably want different per group limit. So we might have to make it per cgroup per device limit.
From the viewpoint of implementation, there is a difficulty in my mind to
implement per cgroup per device limit arising from that io_group is allocated when associated device is firstly used. I guess Gui chose per cgroup limit on all devices approach because of this, right?
Quoted 11 lines
- There does not seem to be any checks for making sure that children cgroups don't have more request descriptors allocated than parent group.
- I am re-thinking that what's the advantage of configuring request descriptors also through cgroups. It does bring in additional complexity with it and it should justfiy the advantages. Can you think of some?
Until and unless we can come up with some significant advantages, I will prefer to continue to use per group limit through
q->nr_group_requestsinterface instead of cgroup. Once things stablize, we can revisit it and see how this interface can be improved.
I agree. I will try to clarify if per group per device limitation is needed or not (or, if it has the advantage beyond the complexity) through some tests.
Tnaks a lot,
Muuhh
Quoted 340 lines
Thanks
VivekQuoted 336 lines
Signed-off-by Gui Jianfeng<[email protected]>--- block/blk-core.c | 23 ++++++++++-- block/blk-settings.c | 1 - block/blk-sysfs.c | 43 ----------------------- block/elevator-fq.c | 94 ++++++++++++++++++++++++++++++++++++++++++++++--- block/elevator-fq.h | 4 ++ 5 files changed, 111 insertions(+), 54 deletions(-)block/blk-core.c +19
-4@@ -722,13 +722,20 @@ static void ioc_set_batching(struct request_queue *q, struct io_context *ioc)static void __freed_request (struct request_queue *q,int sync,struct request_list *rl) { +struct io_group *iog; +unsigned long nr_group_requests; +if (q->rq_data.count[sync]<queue_congestion_off_threshold (q))blk_clear_queue_congested (q, sync);if (q->rq_data.count[sync] +1 <= q->nr_requests)blk_clear_queue_full (q, sync); -if (rl->count[sync] +1 <= q->nr_group_requests) { + iog =rl_iog (rl); + + nr_group_requests =get_group_requests (q, iog); + +if (nr_group_requests&& rl->count[sync] +1 <= nr_group_requests) {if (waitqueue_active (&rl->wait[sync]))wake_up (&rl->wait[sync]); }@@ -828,6 +835,8 @@ static struct request *get_request(struct request_queue *q, int rw_flags,const bool is_sync =rw_is_sync (rw_flags) !=0 ;int may_queue, priv;int sleep_on_global =0 ; +struct io_group *iog; +unsigned long nr_group_requests; may_queue =elv_may_queue (q, rw_flags);if (may_queue ==ELV_MQUEUE_NO )@@ -843,7 +852,12 @@ static struct request *get_request(struct request_queue *q, int rw_flags,if (q->rq_data.count[is_sync]+1 >= q->nr_requests)blk_set_queue_full (q, is_sync); -if (rl->count[is_sync]+1 >= q->nr_group_requests) { + iog =rl_iog (rl); + + nr_group_requests =get_group_requests (q, iog); + +if (nr_group_requests&& + rl->count[is_sync]+1 >= nr_group_requests) { ioc =current_io_context (GFP_ATOMIC , q->node);/* * The queue request descriptor group will fill after this @@ -852,7 +866,7 @@ static struct request *get_request(struct request_queue *q, int rw_flags,* This process will be allowed to complete a batch of * requests, others will be blocked. */ -if (rl->count[is_sync]<= q->nr_group_requests) +if (rl->count[is_sync]<= nr_group_requests)ioc_set_batching (q, ioc);else { if (may_queue != ELV_MQUEUE_MUST@@ -898,7 +912,8 @@ static struct request *get_request(struct request_queue *q, int rw_flags,* from per group request list */ -if (rl->count[is_sync]>= (3 * q->nr_group_requests /2 )) +if (nr_group_requests&& + rl->count[is_sync]>= (3 * nr_group_requests /2 )) goto out; rl->starved[is_sync] =0 ;block/blk-settings.c +0
-1@@ -148,7 +148,6 @@ void blk_queue_make_request(struct request_queue *q, make_request_fn *mfn)* set defaults */ q->nr_requests =BLKDEV_MAX_RQ ; - q->nr_group_requests =BLKDEV_MAX_GROUP_RQ ; q->make_request_fn = mfn;blk_queue_dma_alignment (q,511 );block/blk-sysfs.c +0
-43@@ -78,40 +78,8 @@ queue_requests_store(struct request_queue *q, const char *page, size_t count)return ret; }#ifdef CONFIG_GROUP_IOSCHED -static ssize_t queue_group_requests_show (struct request_queue *q,char *page) -{ -return queue_var_show (q->nr_group_requests, (page)); -} -extern void elv_io_group_congestion_threshold (struct request_queue *q,struct io_group *iog); - -static ssize_t -queue_group_requests_store (struct request_queue *q,const char *page, -size_t count) -{ -struct hlist_node *n; -struct io_group *iog; -struct elv_fq_data *efqd; -unsigned long nr; -int ret =queue_var_store (&nr, page, count); - -if (nr<BLKDEV_MIN_RQ ) - nr =BLKDEV_MIN_RQ ; - -spin_lock_irq (q->queue_lock); - - q->nr_group_requests = nr; - - efqd =&q->elevator->efqd; - -hlist_for_each_entry (iog, n,&efqd->group_list, elv_data_node) { -elv_io_group_congestion_threshold (q, iog); - } - -spin_unlock_irq (q->queue_lock); -return ret; -}#endif static ssize_t queue_ra_show (struct request_queue *q,char *page)@@ -278,14 +246,6 @@ static struct queue_sysfs_entry queue_requests_entry = { .store = queue_requests_store, }; -#ifdef CONFIG_GROUP_IOSCHED -static struct queue_sysfs_entry queue_group_requests_entry = { - .attr = {.name ="nr_group_requests" , .mode =S_IRUGO |S_IWUSR }, - .show = queue_group_requests_show, - .store = queue_group_requests_store, -}; -#endif -static struct queue_sysfs_entry queue_ra_entry = { .attr = {.name ="read_ahead_kb" , .mode =S_IRUGO |S_IWUSR }, .show = queue_ra_show,@@ -360,9 +320,6 @@ static struct queue_sysfs_entry queue_iostats_entry = {static struct attribute *default_attrs[] = { &queue_requests_entry.attr, -#ifdef CONFIG_GROUP_IOSCHED - &queue_group_requests_entry.attr, -#endif &queue_ra_entry.attr, &queue_max_hw_sectors_entry.attr, &queue_max_sectors_entry.attr,block/elevator-fq.c +88
-6@@ -59,6 +59,35 @@ elv_release_ioq(struct elevator_queue *eq, struct io_queue **ioq_ptr);#define for_each_entity_safe (entity, parent) \ for (; entity&& ({ parent = entity->parent; 1; }); entity = parent) +unsigned short get_group_requests (struct request_queue *q, +struct io_group *iog) +{ +struct cgroup_subsys_state *css; +struct io_cgroup *iocg; +unsigned long nr_group_requests; + +if (!iog) +return q->nr_requests; + +rcu_read_lock (); + +if (!iog->iocg_id) { + nr_group_requests =0 ; + goto out; + } + + css =css_lookup (&io_subsys, iog->iocg_id); +if (!css) { + nr_group_requests =0 ; + goto out; + } + + iocg = container_of(css, struct io_cgroup, css); + nr_group_requests = iocg->nr_group_requests; +out: +rcu_read_unlock (); +return nr_group_requests; +}static struct io_entity *bfq_lookup_next_entity (struct io_sched_data *sd,int extract);@@ -1257,14 +1286,17 @@ void elv_io_group_congestion_threshold(struct request_queue *q, struct io_group *iog) {int nr; +unsigned long nr_group_requests; - nr = q->nr_group_requests - (q->nr_group_requests /8 ) +1 ; -if (nr> q->nr_group_requests) - nr = q->nr_group_requests; + nr_group_requests =get_group_requests (q, iog); + + nr = nr_group_requests - (nr_group_requests /8 ) +1 ; +if (nr> nr_group_requests) + nr = nr_group_requests; iog->nr_congestion_on = nr; - nr = q->nr_group_requests - (q->nr_group_requests /8 ) - - (q->nr_group_requests /16 ) -1 ; + nr = nr_group_requests - (nr_group_requests /8 ) + - (nr_group_requests /16 ) -1 ;if (nr<1 ) nr =1 ; iog->nr_congestion_off = nr;@@ -1283,6 +1315,7 @@ int elv_io_group_congested(struct request_queue *q, struct page *page, int sync) {struct io_group *iog;int ret =0 ; +unsigned long nr_group_requests;rcu_read_lock ();@@ -1300,10 +1333,11 @@ int elv_io_group_congested(struct request_queue *q, struct page *page, int sync) } ret =elv_is_iog_congested (q, iog, sync); + nr_group_requests =get_group_requests (q, iog);if (ret)elv_log_iog (&q->elevator->efqd, iog,"iog congested=%d sync=%d" " rl.count[sync]=%d nr_group_requests=%d" , - ret, sync, iog->rl.count[sync], q->nr_group_requests); + ret, sync, iog->rl.count[sync], nr_group_requests);rcu_read_unlock ();return ret; }@@ -1549,6 +1583,48 @@ free_buf:return ret; } +static u64 io_cgroup_nr_requests_read (struct cgroup *cgroup, +struct cftype *cftype) +{ +struct io_cgroup *iocg; +u64 ret; + +if (!cgroup_lock_live_group (cgroup)) +return -ENODEV ; + + iocg =cgroup_to_io_cgroup (cgroup); +spin_lock_irq (&iocg->lock); + ret = iocg->nr_group_requests; +spin_unlock_irq (&iocg->lock); + +cgroup_unlock (); + +return ret; +} + +static int io_cgroup_nr_requests_write (struct cgroup *cgroup, +struct cftype *cftype, +u64 val) +{ +struct io_cgroup *iocg; + +if (val<BLKDEV_MIN_RQ ) + val =BLKDEV_MIN_RQ ; + +if (!cgroup_lock_live_group (cgroup)) +return -ENODEV ; + + iocg =cgroup_to_io_cgroup (cgroup); + +spin_lock_irq (&iocg->lock); + iocg->nr_group_requests = (unsigned long )val; +spin_unlock_irq (&iocg->lock); + +cgroup_unlock (); + +return 0 ; +} +#define SHOW_FUNCTION (__VAR) \ static u64 io_cgroup_##__VAR##_read(struct cgroup *cgroup, \ struct cftype *cftype) \@@ -1735,6 +1811,11 @@ static int io_cgroup_disk_dequeue_read(struct cgroup *cgroup,struct cftype bfqio_files[] = { { + .name ="nr_group_requests" , + .read_u64 = io_cgroup_nr_requests_read, + .write_u64 = io_cgroup_nr_requests_write, + }, + { .name ="policy" , .read_seq_string = io_cgroup_policy_read, .write_string = io_cgroup_policy_write,@@ -1790,6 +1871,7 @@ static struct cgroup_subsys_state *iocg_create(struct cgroup_subsys *subsys,spin_lock_init (&iocg->lock);INIT_HLIST_HEAD (&iocg->group_data); + iocg->nr_group_requests =BLKDEV_MAX_GROUP_RQ ; iocg->weight =IO_DEFAULT_GRP_WEIGHT ; iocg->ioprio_class =IO_DEFAULT_GRP_CLASS ;INIT_LIST_HEAD (&iocg->policy_list);block/elevator-fq.h +4
-1@@ -308,6 +308,7 @@ struct io_cgroup {unsigned int weight;unsigned short ioprio_class; +unsigned long nr_group_requests;/* list of io_policy_node */ struct list_head policy_list;@@ -386,6 +387,9 @@ struct elv_fq_data {unsigned int fairness; }; +extern unsigned short get_group_requests (struct request_queue *q, +struct io_group *iog); +/* Logging facilities. */ #ifdef CONFIG_DEBUG_GROUP_IOSCHED #define elv_log_ioq(efqd, ioq, fmt, args...) \ --1.5.4.rc3
Signature · 3 lines
IKEDA, Munehiro
NEC Corporation of America
m-ikeda-MDRzhb/z0dd8UrSeD/[email protected]Re: [PATCH] io-controller: implement per group request allocation limitation
In reply to Vivek GoyalQuoted 8 lines · Vivek Goyal
Quoted 6 lines · Gui Jianfeng
Hi Vivek,
This patch exports a cgroup based per group request limits interface. and removes the global one. Now we can use this interface to perform different request allocation limitation for different groups.
Quoted 6 lines
Thanks Gui. Few points come to mind.
- You seem to be making this as per cgroup limit on all devices. I guess that different devices in the system can have different settings of
q->nr_requestsand hence will probably want different per group limit. So we might have to make it per cgroup per device limit.
Yes, per cgroup per device limitation seems more reasonable. I'll see what i can do.
Quoted 3 lines
- There does not seem to be any checks for making sure that children cgroups don't have more request descriptors allocated than parent group.
Do we really need to make it hierarchical? IMHO, maintaining this limitation for cgroups independently is enough.
Quoted 4 lines
- I am re-thinking that what's the advantage of configuring request descriptors also through cgroups. It does bring in additional complexity with it and it should justfiy the advantages. Can you think of some?
I'll try, but at least, this feature lets us be able to do more accurate limitation. :)
Quoted 5 lines
Until and unless we can come up with some significant advantages, I will prefer to continue to use per group limit through
q->nr_group_requestsinterface instead of cgroup. Once things stablize, we can revisit it and see how this interface can be improved.
I agree.
Signature · 2 lines
Regards Gui Jianfeng
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: [PATCH] io-controller: implement per group request allocation limitation
Quoted 6 lines · Gui Jianfeng
Hi Vivek,
This patch exports a cgroup based per group request limits interface. and removes the global one. Now we can use this interface to perform different request allocation limitation for different groups.
Thanks Gui. Few points come to mind.
- You seem to be making this as per cgroup limit on all devices. I guess that different devices in the system can have different settings of q->nr_requests and hence will probably want different per group limit. So we might have to make it per cgroup per device limit.
- There does not seem to be any checks for making sure that children cgroups don't have more request descriptors allocated than parent group.
- I am re-thinking that what's the advantage of configuring request descriptors also through cgroups. It does bring in additional complexity with it and it should justfiy the advantages. Can you think of some?
Until and unless we can come up with some significant advantages, I will prefer to continue to use per group limit through q->nr_group_requests interface instead of cgroup. Once things stablize, we can revisit it and see how this interface can be improved.
Thanks
Vivek
Quoted 336 lines
Signed-off-by Gui Jianfeng <[email protected]>--- block/blk-core.c | 23 ++++++++++-- block/blk-settings.c | 1 - block/blk-sysfs.c | 43 ----------------------- block/elevator-fq.c | 94 ++++++++++++++++++++++++++++++++++++++++++++++--- block/elevator-fq.h | 4 ++ 5 files changed, 111 insertions(+), 54 deletions(-)block/blk-core.c +19
-4@@ -722,13 +722,20 @@ static void ioc_set_batching(struct request_queue *q, struct io_context *ioc)static void __freed_request (struct request_queue *q,int sync,struct request_list *rl) { +struct io_group *iog; +unsigned long nr_group_requests; +if (q->rq_data.count[sync] <queue_congestion_off_threshold (q))blk_clear_queue_congested (q, sync);if (q->rq_data.count[sync] +1 <= q->nr_requests)blk_clear_queue_full (q, sync); -if (rl->count[sync] +1 <= q->nr_group_requests) { + iog =rl_iog (rl); + + nr_group_requests =get_group_requests (q, iog); + +if (nr_group_requests && rl->count[sync] +1 <= nr_group_requests) {if (waitqueue_active (&rl->wait[sync]))wake_up (&rl->wait[sync]); }@@ -828,6 +835,8 @@ static struct request *get_request(struct request_queue *q, int rw_flags,const bool is_sync =rw_is_sync (rw_flags) !=0 ;int may_queue, priv;int sleep_on_global =0 ; +struct io_group *iog; +unsigned long nr_group_requests; may_queue =elv_may_queue (q, rw_flags);if (may_queue ==ELV_MQUEUE_NO )@@ -843,7 +852,12 @@ static struct request *get_request(struct request_queue *q, int rw_flags,if (q->rq_data.count[is_sync]+1 >= q->nr_requests)blk_set_queue_full (q, is_sync); -if (rl->count[is_sync]+1 >= q->nr_group_requests) { + iog =rl_iog (rl); + + nr_group_requests =get_group_requests (q, iog); + +if (nr_group_requests && + rl->count[is_sync]+1 >= nr_group_requests) { ioc =current_io_context (GFP_ATOMIC , q->node);/* * The queue request descriptor group will fill after this @@ -852,7 +866,7 @@ static struct request *get_request(struct request_queue *q, int rw_flags,* This process will be allowed to complete a batch of * requests, others will be blocked. */ -if (rl->count[is_sync] <= q->nr_group_requests) +if (rl->count[is_sync] <= nr_group_requests)ioc_set_batching (q, ioc);else { if (may_queue != ELV_MQUEUE_MUST@@ -898,7 +912,8 @@ static struct request *get_request(struct request_queue *q, int rw_flags,* from per group request list */ -if (rl->count[is_sync] >= (3 * q->nr_group_requests /2 )) +if (nr_group_requests && + rl->count[is_sync] >= (3 * nr_group_requests /2 )) goto out; rl->starved[is_sync] =0 ;block/blk-settings.c +0
-1@@ -148,7 +148,6 @@ void blk_queue_make_request(struct request_queue *q, make_request_fn *mfn)* set defaults */ q->nr_requests =BLKDEV_MAX_RQ ; - q->nr_group_requests =BLKDEV_MAX_GROUP_RQ ; q->make_request_fn = mfn;blk_queue_dma_alignment (q,511 );block/blk-sysfs.c +0
-43@@ -78,40 +78,8 @@ queue_requests_store(struct request_queue *q, const char *page, size_t count)return ret; }#ifdef CONFIG_GROUP_IOSCHED -static ssize_t queue_group_requests_show (struct request_queue *q,char *page) -{ -return queue_var_show (q->nr_group_requests, (page)); -} -extern void elv_io_group_congestion_threshold (struct request_queue *q,struct io_group *iog); - -static ssize_t -queue_group_requests_store (struct request_queue *q,const char *page, -size_t count) -{ -struct hlist_node *n; -struct io_group *iog; -struct elv_fq_data *efqd; -unsigned long nr; -int ret =queue_var_store (&nr, page, count); - -if (nr <BLKDEV_MIN_RQ ) - nr =BLKDEV_MIN_RQ ; - -spin_lock_irq (q->queue_lock); - - q->nr_group_requests = nr; - - efqd = &q->elevator->efqd; - -hlist_for_each_entry (iog, n, &efqd->group_list, elv_data_node) { -elv_io_group_congestion_threshold (q, iog); - } - -spin_unlock_irq (q->queue_lock); -return ret; -}#endif static ssize_t queue_ra_show (struct request_queue *q,char *page)@@ -278,14 +246,6 @@ static struct queue_sysfs_entry queue_requests_entry = { .store = queue_requests_store, }; -#ifdef CONFIG_GROUP_IOSCHED -static struct queue_sysfs_entry queue_group_requests_entry = { - .attr = {.name ="nr_group_requests" , .mode =S_IRUGO |S_IWUSR }, - .show = queue_group_requests_show, - .store = queue_group_requests_store, -}; -#endif -static struct queue_sysfs_entry queue_ra_entry = { .attr = {.name ="read_ahead_kb" , .mode =S_IRUGO |S_IWUSR }, .show = queue_ra_show,@@ -360,9 +320,6 @@ static struct queue_sysfs_entry queue_iostats_entry = {static struct attribute *default_attrs[] = { &queue_requests_entry.attr, -#ifdef CONFIG_GROUP_IOSCHED - &queue_group_requests_entry.attr, -#endif &queue_ra_entry.attr, &queue_max_hw_sectors_entry.attr, &queue_max_sectors_entry.attr,block/elevator-fq.c +88
-6@@ -59,6 +59,35 @@ elv_release_ioq(struct elevator_queue *eq, struct io_queue **ioq_ptr);#define for_each_entity_safe (entity, parent) \ for (; entity && ({ parent = entity->parent; 1; }); entity = parent) +unsigned short get_group_requests (struct request_queue *q, +struct io_group *iog) +{ +struct cgroup_subsys_state *css; +struct io_cgroup *iocg; +unsigned long nr_group_requests; + +if (!iog) +return q->nr_requests; + +rcu_read_lock (); + +if (!iog->iocg_id) { + nr_group_requests =0 ; + goto out; + } + + css =css_lookup (&io_subsys, iog->iocg_id); +if (!css) { + nr_group_requests =0 ; + goto out; + } + + iocg = container_of(css, struct io_cgroup, css); + nr_group_requests = iocg->nr_group_requests; +out: +rcu_read_unlock (); +return nr_group_requests; +}static struct io_entity *bfq_lookup_next_entity (struct io_sched_data *sd,int extract);@@ -1257,14 +1286,17 @@ void elv_io_group_congestion_threshold(struct request_queue *q, struct io_group *iog) {int nr; +unsigned long nr_group_requests; - nr = q->nr_group_requests - (q->nr_group_requests /8 ) +1 ; -if (nr > q->nr_group_requests) - nr = q->nr_group_requests; + nr_group_requests =get_group_requests (q, iog); + + nr = nr_group_requests - (nr_group_requests /8 ) +1 ; +if (nr > nr_group_requests) + nr = nr_group_requests; iog->nr_congestion_on = nr; - nr = q->nr_group_requests - (q->nr_group_requests /8 ) - - (q->nr_group_requests /16 ) -1 ; + nr = nr_group_requests - (nr_group_requests /8 ) + - (nr_group_requests /16 ) -1 ;if (nr <1 ) nr =1 ; iog->nr_congestion_off = nr;@@ -1283,6 +1315,7 @@ int elv_io_group_congested(struct request_queue *q, struct page *page, int sync) {struct io_group *iog;int ret =0 ; +unsigned long nr_group_requests;rcu_read_lock ();@@ -1300,10 +1333,11 @@ int elv_io_group_congested(struct request_queue *q, struct page *page, int sync) } ret =elv_is_iog_congested (q, iog, sync); + nr_group_requests =get_group_requests (q, iog);if (ret)elv_log_iog (&q->elevator->efqd, iog,"iog congested=%d sync=%d" " rl.count[sync]=%d nr_group_requests=%d" , - ret, sync, iog->rl.count[sync], q->nr_group_requests); + ret, sync, iog->rl.count[sync], nr_group_requests);rcu_read_unlock ();return ret; }@@ -1549,6 +1583,48 @@ free_buf:return ret; } +static u64 io_cgroup_nr_requests_read (struct cgroup *cgroup, +struct cftype *cftype) +{ +struct io_cgroup *iocg; +u64 ret; + +if (!cgroup_lock_live_group (cgroup)) +return -ENODEV ; + + iocg =cgroup_to_io_cgroup (cgroup); +spin_lock_irq (&iocg->lock); + ret = iocg->nr_group_requests; +spin_unlock_irq (&iocg->lock); + +cgroup_unlock (); + +return ret; +} + +static int io_cgroup_nr_requests_write (struct cgroup *cgroup, +struct cftype *cftype, +u64 val) +{ +struct io_cgroup *iocg; + +if (val <BLKDEV_MIN_RQ ) + val =BLKDEV_MIN_RQ ; + +if (!cgroup_lock_live_group (cgroup)) +return -ENODEV ; + + iocg =cgroup_to_io_cgroup (cgroup); + +spin_lock_irq (&iocg->lock); + iocg->nr_group_requests = (unsigned long )val; +spin_unlock_irq (&iocg->lock); + +cgroup_unlock (); + +return 0 ; +} +#define SHOW_FUNCTION (__VAR) \ static u64 io_cgroup_##__VAR##_read(struct cgroup *cgroup, \ struct cftype *cftype) \@@ -1735,6 +1811,11 @@ static int io_cgroup_disk_dequeue_read(struct cgroup *cgroup,struct cftype bfqio_files[] = { { + .name ="nr_group_requests" , + .read_u64 = io_cgroup_nr_requests_read, + .write_u64 = io_cgroup_nr_requests_write, + }, + { .name ="policy" , .read_seq_string = io_cgroup_policy_read, .write_string = io_cgroup_policy_write,@@ -1790,6 +1871,7 @@ static struct cgroup_subsys_state *iocg_create(struct cgroup_subsys *subsys,spin_lock_init (&iocg->lock);INIT_HLIST_HEAD (&iocg->group_data); + iocg->nr_group_requests =BLKDEV_MAX_GROUP_RQ ; iocg->weight =IO_DEFAULT_GRP_WEIGHT ; iocg->ioprio_class =IO_DEFAULT_GRP_CLASS ;INIT_LIST_HEAD (&iocg->policy_list);block/elevator-fq.h +4
-0@@ -308,6 +308,7 @@ struct io_cgroup {unsigned int weight;unsigned short ioprio_class; +unsigned long nr_group_requests;/* list of io_policy_node */ struct list_head policy_list;@@ -386,6 +387,9 @@ struct elv_fq_data {unsigned int fairness; }; +extern unsigned short get_group_requests (struct request_queue *q, +struct io_group *iog); +/* Logging facilities. */ #ifdef CONFIG_DEBUG_GROUP_IOSCHED #define elv_log_ioq(efqd, ioq, fmt, args...) \Signature · 1 line
1.5.4.rc3
Re: [PATCH] io-controller: implement per group request allocation limitation
In reply to a message not in the archiveGui Jianfeng wrote, on 07/09/2009 09:56 PM:
Quoted 8 lines
Hi Vivek,
This patch exports a cgroup based per group request limits interface. and removes the global one. Now we can use this interface to perform different request allocation limitation for different groups.
Signed-off-by Gui Jianfeng<[email protected]>---
(snip)
Hi Jianfeng,
If this helps.
If calling elv_io_group_congestion_threshold() before setting iog->iocg_id, iocg->nr_group_requests cannot be referred. As a result of it, iog->nr_congestion_on is always misculculated as 0.
This patch moves the calling of elv_io_group_congestion_threshold() after setting iog->iocg_id.
--- block/elevator-fq.c | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-)
block/elevator-fq.c +2 -1
@@ -2195,7 +2195,6 @@ static struct io_group *io_alloc_root_group(struct request_queue *q, iog->sched_data.service_tree[i] =IO_SERVICE_TREE_INIT ;blk_init_request_list (&iog->rl); -elv_io_group_congestion_threshold (q, iog); iocg = &io_root_cgroup;spin_lock_irq (&iocg->lock);
@@ -2204,6 +2203,8 @@ static struct io_group *io_alloc_root_group(struct request_queue *q, iog->iocg_id =css_id (&iocg->css);spin_unlock_irq (&iocg->lock); +elv_io_group_congestion_threshold (q, iog); +#ifdef CONFIG_DEBUG_GROUP_IOSCHED io_group_path (iog, iog->path,sizeof (iog->path));#endif
Signature · 9 lines
1.6.2.5
--
IKEDA, Munehiro
NEC Corporation of America
m-ikeda-MDRzhb/z0dd8UrSeD/[email protected]Re: [PATCH] io-controller: implement per group request allocation limitation
In reply to a message not in the archiveGui Jianfeng wrote, on 07/09/2009 09:56 PM:
Quoted 7 lines
Hi Vivek,
This patch exports a cgroup based per group request limits interface. and removes the global one. Now we can use this interface to perform different request allocation limitation for different groups.
Signed-off-by Gui Jianfeng<[email protected]>
(snip)
Hi Jianfeng,
If this helps, again.
A patch posted from Gui Jianfeng on 2009/07/09 adds per-cgroup nr_requests control by io.nr_group_requests. The patch missed to update iog->nr_congestion_{on|off} and this patch adds the missing-link.
--- block/elevator-fq.c | 11 +++++++++++ 1 files changed, 11 insertions(+), 0 deletions(-)
block/elevator-fq.c +11 -0
@@ -1607,6 +1607,10 @@ static int io_cgroup_nr_requests_write(struct cgroup *cgroup, u64 val) {struct io_cgroup *iocg; +struct io_group *iog; +struct elv_fq_data *efqd; +struct request_queue *q; +struct hlist_node *n;if (val <BLKDEV_MIN_RQ ) val =BLKDEV_MIN_RQ ;
@@ -1618,6 +1622,13 @@ static int io_cgroup_nr_requests_write(struct cgroup *cgroup,spin_lock_irq (&iocg->lock); iocg->nr_group_requests = (unsigned long )val; +hlist_for_each_entry (iog, n, &iocg->group_data, group_node) { +rcu_read_lock (); + efqd =rcu_dereference (iog->key); + q = efqd->queue; +rcu_read_unlock (); +elv_io_group_congestion_threshold (q, iog); + }spin_unlock_irq (&iocg->lock);cgroup_unlock ();
Signature · 7 lines
1.6.2.5
--
IKEDA, Munehiro
NEC Corporation of America
m-ikeda-MDRzhb/z0dd8UrSeD/[email protected]Re: [PATCH] io-controller: implement per group request allocation limitation
In reply to Gui JianfengGui Jianfeng wrote, on 07/09/2009 09:56 PM:
Quoted 8 lines
Hi Vivek,
This patch exports a cgroup based per group request limits interface. and removes the global one. Now we can use this interface to perform different request allocation limitation for different groups.
Signed-off-by Gui Jianfeng<[email protected]>---
(snip)
Hi Jianfeng,
If this helps.
If calling elv_io_group_congestion_threshold() before setting iog->iocg_id, iocg->nr_group_requests cannot be referred. As a result of it, iog->nr_congestion_on is always misculculated as 0.
This patch moves the calling of elv_io_group_congestion_threshold() after setting iog->iocg_id.
--- block/elevator-fq.c | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-)
block/elevator-fq.c +2 -1
@@ -2195,7 +2195,6 @@ static struct io_group *io_alloc_root_group(struct request_queue *q, iog->sched_data.service_tree[i] =IO_SERVICE_TREE_INIT ;blk_init_request_list (&iog->rl); -elv_io_group_congestion_threshold (q, iog); iocg = &io_root_cgroup;spin_lock_irq (&iocg->lock);
@@ -2204,6 +2203,8 @@ static struct io_group *io_alloc_root_group(struct request_queue *q, iog->iocg_id =css_id (&iocg->css);spin_unlock_irq (&iocg->lock); +elv_io_group_congestion_threshold (q, iog); +#ifdef CONFIG_DEBUG_GROUP_IOSCHED io_group_path (iog, iog->path,sizeof (iog->path));#endif
Signature · 9 lines
1.6.2.5
--
IKEDA, Munehiro
NEC Corporation of America
[email protected]Re: [PATCH] io-controller: implement per group request allocation limitation
Quoted 23 lines · Munehiro Ikeda
Gui Jianfeng wrote, on 07/09/2009 09:56 PM:
Quoted 8 lines
Hi Vivek,
This patch exports a cgroup based per group request limits interface. and removes the global one. Now we can use this interface to perform different request allocation limitation for different groups.
Signed-off-by Gui Jianfeng<[email protected]>---
(snip)
Hi Jianfeng,
If this helps.
If calling
elv_io_group_congestion_threshold()before settingiog->iocg_id,iocg->nr_group_requestscannot be referred. As a result of it,iog->nr_congestion_onis always misculculated as 0.
This patch moves the calling ofelv_io_group_congestion_threshold()after settingiog->iocg_id.
Quoted 1 line
Signed-off-by Munehiro "Muuhh" Ikeda <[email protected]>
That's true, thanks.
Signature · 2 lines
Regards Gui Jianfeng
A message that is not in the archive (sent off-list or not yet received): 4A7796D2.4030104-MDRzhb/z0dd8UrSeD/[email protected]
Re: [PATCH] io-controller: implement per group request allocation limitation
Quoted 23 lines · Munehiro Ikeda
Gui Jianfeng wrote, on 07/09/2009 09:56 PM:
Quoted 8 lines
Hi Vivek,
This patch exports a cgroup based per group request limits interface. and removes the global one. Now we can use this interface to perform different request allocation limitation for different groups.
Signed-off-by Gui Jianfeng<[email protected]>---
(snip)
Hi Jianfeng,
If this helps.
If calling
elv_io_group_congestion_threshold()before settingiog->iocg_id,iocg->nr_group_requestscannot be referred. As a result of it,iog->nr_congestion_onis always misculculated as 0.
This patch moves the calling ofelv_io_group_congestion_threshold()after settingiog->iocg_id.
Quoted 1 line
That's true, thanks.
Signature · 2 lines
Regards Gui Jianfeng
Re: [PATCH] io-controller: implement per group request allocation limitation
In reply to Gui JianfengGui Jianfeng wrote, on 07/09/2009 09:56 PM:
Quoted 7 lines
Hi Vivek,
This patch exports a cgroup based per group request limits interface. and removes the global one. Now we can use this interface to perform different request allocation limitation for different groups.
Signed-off-by Gui Jianfeng<[email protected]>
(snip)
Hi Jianfeng,
If this helps, again.
A patch posted from Gui Jianfeng on 2009/07/09 adds per-cgroup nr_requests control by io.nr_group_requests. The patch missed to update iog->nr_congestion_{on|off} and this patch adds the missing-link.
--- block/elevator-fq.c | 11 +++++++++++ 1 files changed, 11 insertions(+), 0 deletions(-)
block/elevator-fq.c +11 -0
@@ -1607,6 +1607,10 @@ static int io_cgroup_nr_requests_write(struct cgroup *cgroup, u64 val) {struct io_cgroup *iocg; +struct io_group *iog; +struct elv_fq_data *efqd; +struct request_queue *q; +struct hlist_node *n;if (val <BLKDEV_MIN_RQ ) val =BLKDEV_MIN_RQ ;
@@ -1618,6 +1622,13 @@ static int io_cgroup_nr_requests_write(struct cgroup *cgroup,spin_lock_irq (&iocg->lock); iocg->nr_group_requests = (unsigned long )val; +hlist_for_each_entry (iog, n, &iocg->group_data, group_node) { +rcu_read_lock (); + efqd =rcu_dereference (iog->key); + q = efqd->queue; +rcu_read_unlock (); +elv_io_group_congestion_threshold (q, iog); + }spin_unlock_irq (&iocg->lock);cgroup_unlock ();
Signature · 7 lines
1.6.2.5
--
IKEDA, Munehiro
NEC Corporation of America
[email protected]Re: [PATCH] io-controller: implement per group request allocation limitation
Quoted 12 lines · Munehiro Ikeda
Gui Jianfeng wrote, on 07/09/2009 09:56 PM:
Quoted 7 lines
Hi Vivek,
This patch exports a cgroup based per group request limits interface. and removes the global one. Now we can use this interface to perform different request allocation limitation for different groups.
Signed-off-by Gui Jianfeng<[email protected]>(snip)
Hi Jianfeng,
Quoted 7 lines
If this helps, again.
A patch posted from Gui Jianfeng on 2009/07/09 adds per-cgroup
nr_requestscontrol by io.nr_group_requests. The patch missed to update iog->nr_congestion_{on|off} and this patch adds the missing-link.Signed-off-by Munehiro "Muuhh" Ikeda <[email protected]>
Yes, i'v fixed this issue for the global nr_group_requests updating in V6, but forgot to update it in this patch. Thanks :)
Signature · 2 lines
Regards Gui Jianfeng
A message that is not in the archive (sent off-list or not yet received): 4A779719.1070900-MDRzhb/z0dd8UrSeD/[email protected]
Re: [PATCH] io-controller: implement per group request allocation limitation
Quoted 12 lines · Munehiro Ikeda
Gui Jianfeng wrote, on 07/09/2009 09:56 PM:
Quoted 7 lines
Hi Vivek,
This patch exports a cgroup based per group request limits interface. and removes the global one. Now we can use this interface to perform different request allocation limitation for different groups.
Signed-off-by Gui Jianfeng<[email protected]>(snip)
Hi Jianfeng,
Quoted 7 lines
If this helps, again.
A patch posted from Gui Jianfeng on 2009/07/09 adds per-cgroup
nr_requestscontrol by io.nr_group_requests. The patch missed to update iog->nr_congestion_{on|off} and this patch adds the missing-link.
Yes, i'v fixed this issue for the global nr_group_requests updating in V6, but forgot to update it in this patch. Thanks :)
Signature · 2 lines
Regards Gui Jianfeng
Hi,
Here are some fio test results for IO Controller V6 built and not built. Iozone test results are also attached.
Arch: X86
Mem: 1G
Disk: 320G
IO Scheduler: CFQ
============
By normal read and write syscall.
Block Size:32K
File Size: 1G * 10
Mode Normal read | Random read | Normal write | Random write | Direct read | Direct write
2.6.31-rc1 47,932KiB/s 3,566KiB/s 45,693KiB/s 8,501KiB/s 50,088KiB/s 43,473KiB/s
2.6.31-rc1-Vivek-V6 47,231KiB/s 3,411KiB/s 42,451KiB/s 8,714KiB/s 51,284KiB/s 42,341KiB/s
Performance -1.5% -4.4% -7.0% +2.5% +2.4% -2.6%
============
By mmap.
Block Size:32K
File Size: 500M
Mode Normal read | Random read | Normal write | Random write
2.6.31-rc1 49,951KiB/s 3,245KiB/s 21,950KiB/s 2,771KiB/s
2.6.31-rc1-Vivek-V6 49,951KiB/s 3,154KiB/s 22,593KiB/s 2,648KiB/s
Performance 0% -2.8% +2.9% -4.4%
============
By libaio calls
Block Size:32K
File Size: 500M
Mode Normal read | Random read | Normal write | Random write
2.6.31-rc1 49,447KiB/s 3,296KiB/s 57,519KiB/s 21,093KiB/s
2.6.31-rc1-Vivek-V6 50,142KiB/s 3,238KiB/s 57,791KiB/s 21,283KiB/s
Performance +1.4% -1.8% 0% +0.1%
============
Quoted 5 lines · Gui Jianfeng
Hi,
Here are some fio test results for IO Controller V6 built and not built. Iozone test results are also attached.
Hi Gui,
Thanks a lot for some performance numbers. It seems to be a mixed chart. Performance gains at some places and loss at others. I am curious about that -7.0% for normal writes. Not sure what can contribute to that.
What was the value of "fairness" parameter when you ran those tests? Can you please set fairness = 0 and re-run the tests (if you have already not done so).
By default fairness is set to 1 in V6. With fairness = 0, we should be very close to existing CFQ behavior. If not, then we need to dive deeper and see why variations are happening.
Is it also possible to run the same tests with V7.
Thanks
Vivek
Quoted 61 lines
Arch: X86
Mem: 1G
Disk: 320G
IO Scheduler: CFQ============
By normal read and write syscall.Block Size:32K
File Size: 1G * 10Mode Normal read | Random read | Normal write | Random write | Direct read | Direct write
2.6.31-rc1 47,932KiB/s 3,566KiB/s 45,693KiB/s 8,501KiB/s 50,088KiB/s 43,473KiB/s
2.6.31-rc1-Vivek-V6 47,231KiB/s 3,411KiB/s 42,451KiB/s 8,714KiB/s 51,284KiB/s 42,341KiB/s
Performance -1.5% -4.4% -7.0% +2.5% +2.4% -2.6%
============
By mmap.
Block Size:32K
File Size: 500MMode Normal read | Random read | Normal write | Random write
2.6.31-rc1 49,951KiB/s 3,245KiB/s 21,950KiB/s 2,771KiB/s
2.6.31-rc1-Vivek-V6 49,951KiB/s 3,154KiB/s 22,593KiB/s 2,648KiB/s
Performance 0% -2.8% +2.9% -4.4%
============
By libaio calls
Block Size:32K
File Size: 500MMode Normal read | Random read | Normal write | Random write
2.6.31-rc1 49,447KiB/s 3,296KiB/s 57,519KiB/s 21,093KiB/s
2.6.31-rc1-Vivek-V6 50,142KiB/s 3,238KiB/s 57,791KiB/s 21,283KiB/s
Performance +1.4% -1.8% 0% +0.1%
============
Quoted 7 lines · Vivek Goyal
Quoted 5 lines · Gui Jianfeng
Hi,
Here are some fio test results for IO Controller V6 built and not built. Iozone test results are also attached.
Quoted 6 lines
Hi Gui,
Thanks a lot for some performance numbers. It seems to be a mixed chart. Performance gains at some places and loss at others. I am curious about that -7.0% for normal writes. Not sure what can contribute to that.
Gui, can you also try with CONFIG_TRACK_ASYNC_CONTEXT=n and see if it improves buffered write performance.
Thanks
Vivek
Quoted 76 lines
What was the value of "fairness" parameter when you ran those tests? Can you please set fairness = 0 and re-run the tests (if you have already not done so).
By default fairness is set to 1 in V6. With fairness = 0, we should be very close to existing CFQ behavior. If not, then we need to dive deeper and see why variations are happening.
Is it also possible to run the same tests with V7.
Thanks
VivekQuoted 61 lines
Arch: X86
Mem: 1G
Disk: 320G
IO Scheduler: CFQ============
By normal read and write syscall.Block Size:32K
File Size: 1G * 10Mode Normal read | Random read | Normal write | Random write | Direct read | Direct write
2.6.31-rc1 47,932KiB/s 3,566KiB/s 45,693KiB/s 8,501KiB/s 50,088KiB/s 43,473KiB/s
2.6.31-rc1-Vivek-V6 47,231KiB/s 3,411KiB/s 42,451KiB/s 8,714KiB/s 51,284KiB/s 42,341KiB/s
Performance -1.5% -4.4% -7.0% +2.5% +2.4% -2.6%
============
By mmap.
Block Size:32K
File Size: 500MMode Normal read | Random read | Normal write | Random write
2.6.31-rc1 49,951KiB/s 3,245KiB/s 21,950KiB/s 2,771KiB/s
2.6.31-rc1-Vivek-V6 49,951KiB/s 3,154KiB/s 22,593KiB/s 2,648KiB/s
Performance 0% -2.8% +2.9% -4.4%
============
By libaio calls
Block Size:32K
File Size: 500MMode Normal read | Random read | Normal write | Random write
2.6.31-rc1 49,447KiB/s 3,296KiB/s 57,519KiB/s 21,093KiB/s
2.6.31-rc1-Vivek-V6 50,142KiB/s 3,238KiB/s 57,791KiB/s 21,283KiB/s
Performance +1.4% -1.8% 0% +0.1%
============
Quoted 9 lines · Vivek Goyal
Quoted 8 lines · Vivek Goyal
Quoted 5 lines · Gui Jianfeng
Hi,
Here are some fio test results for IO Controller V6 built and not built. Iozone test results are also attached.
Hi Gui,
Quoted 7 lines
Quoted 4 lines
Thanks a lot for some performance numbers. It seems to be a mixed chart. Performance gains at some places and loss at others. I am curious about that -7.0% for normal writes. Not sure what can contribute to that.
Gui, can you also try with
CONFIG_TRACK_ASYNC_CONTEXT=nand see if it improves buffered write performance.
Hi Vivek,
Ok, I'll do it, and i'll do the performance test for IO Controller V7, and post the results as soon as i get them.
Quoted 82 lines
Thanks
VivekQuoted 75 lines
What was the value of "fairness" parameter when you ran those tests? Can you please set fairness = 0 and re-run the tests (if you have already not done so).
By default fairness is set to 1 in V6. With fairness = 0, we should be very close to existing CFQ behavior. If not, then we need to dive deeper and see why variations are happening.
Is it also possible to run the same tests with V7.
Thanks
VivekQuoted 61 lines
Arch: X86
Mem: 1G
Disk: 320G
IO Scheduler: CFQ============
By normal read and write syscall.Block Size:32K
File Size: 1G * 10Mode Normal read | Random read | Normal write | Random write | Direct read | Direct write
2.6.31-rc1 47,932KiB/s 3,566KiB/s 45,693KiB/s 8,501KiB/s 50,088KiB/s 43,473KiB/s
2.6.31-rc1-Vivek-V6 47,231KiB/s 3,411KiB/s 42,451KiB/s 8,714KiB/s 51,284KiB/s 42,341KiB/s
Performance -1.5% -4.4% -7.0% +2.5% +2.4% -2.6%
============
By mmap.
Block Size:32K
File Size: 500MMode Normal read | Random read | Normal write | Random write
2.6.31-rc1 49,951KiB/s 3,245KiB/s 21,950KiB/s 2,771KiB/s
2.6.31-rc1-Vivek-V6 49,951KiB/s 3,154KiB/s 22,593KiB/s 2,648KiB/s
Performance 0% -2.8% +2.9% -4.4%
============
By libaio calls
Block Size:32K
File Size: 500MMode Normal read | Random read | Normal write | Random write
2.6.31-rc1 49,447KiB/s 3,296KiB/s 57,519KiB/s 21,093KiB/s
2.6.31-rc1-Vivek-V6 50,142KiB/s 3,238KiB/s 57,791KiB/s 21,283KiB/s
Performance +1.4% -1.8% 0% +0.1%
============
Signature · 2 lines
Regards Gui Jianfeng
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 9 lines · Vivek Goyal
Quoted 8 lines · Vivek Goyal
Quoted 5 lines · Gui Jianfeng
Hi,
Here are some fio test results for IO Controller V6 built and not built. Iozone test results are also attached.
Hi Gui,
Quoted 7 lines
Quoted 4 lines
Thanks a lot for some performance numbers. It seems to be a mixed chart. Performance gains at some places and loss at others. I am curious about that -7.0% for normal writes. Not sure what can contribute to that.
Gui, can you also try with
CONFIG_TRACK_ASYNC_CONTEXT=nand see if it improves buffered write performance.
Hi Vivek,
Ok, I'll do it, and i'll do the performance test for IO Controller V7, and post the results as soon as i get them.
Quoted 82 lines
Thanks
VivekQuoted 75 lines
What was the value of "fairness" parameter when you ran those tests? Can you please set fairness = 0 and re-run the tests (if you have already not done so).
By default fairness is set to 1 in V6. With fairness = 0, we should be very close to existing CFQ behavior. If not, then we need to dive deeper and see why variations are happening.
Is it also possible to run the same tests with V7.
Thanks
VivekQuoted 61 lines
Arch: X86
Mem: 1G
Disk: 320G
IO Scheduler: CFQ============
By normal read and write syscall.Block Size:32K
File Size: 1G * 10Mode Normal read | Random read | Normal write | Random write | Direct read | Direct write
2.6.31-rc1 47,932KiB/s 3,566KiB/s 45,693KiB/s 8,501KiB/s 50,088KiB/s 43,473KiB/s
2.6.31-rc1-Vivek-V6 47,231KiB/s 3,411KiB/s 42,451KiB/s 8,714KiB/s 51,284KiB/s 42,341KiB/s
Performance -1.5% -4.4% -7.0% +2.5% +2.4% -2.6%
============
By mmap.
Block Size:32K
File Size: 500MMode Normal read | Random read | Normal write | Random write
2.6.31-rc1 49,951KiB/s 3,245KiB/s 21,950KiB/s 2,771KiB/s
2.6.31-rc1-Vivek-V6 49,951KiB/s 3,154KiB/s 22,593KiB/s 2,648KiB/s
Performance 0% -2.8% +2.9% -4.4%
============
By libaio calls
Block Size:32K
File Size: 500MMode Normal read | Random read | Normal write | Random write
2.6.31-rc1 49,447KiB/s 3,296KiB/s 57,519KiB/s 21,093KiB/s
2.6.31-rc1-Vivek-V6 50,142KiB/s 3,238KiB/s 57,791KiB/s 21,283KiB/s
Performance +1.4% -1.8% 0% +0.1%
============
Signature · 7 lines
Regards Gui Jianfeng _______________________________________________ Containers mailing list [email protected] https://lists.linux-foundation.org/mailman/listinfo/containers
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 7 lines · Vivek Goyal
Quoted 5 lines · Gui Jianfeng
Hi,
Here are some fio test results for IO Controller V6 built and not built. Iozone test results are also attached.
Quoted 6 lines
Hi Gui,
Thanks a lot for some performance numbers. It seems to be a mixed chart. Performance gains at some places and loss at others. I am curious about that -7.0% for normal writes. Not sure what can contribute to that.
Gui, can you also try with CONFIG_TRACK_ASYNC_CONTEXT=n and see if it improves buffered write performance.
Thanks
Vivek
Quoted 76 lines
What was the value of "fairness" parameter when you ran those tests? Can you please set fairness = 0 and re-run the tests (if you have already not done so).
By default fairness is set to 1 in V6. With fairness = 0, we should be very close to existing CFQ behavior. If not, then we need to dive deeper and see why variations are happening.
Is it also possible to run the same tests with V7.
Thanks
VivekQuoted 61 lines
Arch: X86
Mem: 1G
Disk: 320G
IO Scheduler: CFQ============
By normal read and write syscall.Block Size:32K
File Size: 1G * 10Mode Normal read | Random read | Normal write | Random write | Direct read | Direct write
2.6.31-rc1 47,932KiB/s 3,566KiB/s 45,693KiB/s 8,501KiB/s 50,088KiB/s 43,473KiB/s
2.6.31-rc1-Vivek-V6 47,231KiB/s 3,411KiB/s 42,451KiB/s 8,714KiB/s 51,284KiB/s 42,341KiB/s
Performance -1.5% -4.4% -7.0% +2.5% +2.4% -2.6%
============
By mmap.
Block Size:32K
File Size: 500MMode Normal read | Random read | Normal write | Random write
2.6.31-rc1 49,951KiB/s 3,245KiB/s 21,950KiB/s 2,771KiB/s
2.6.31-rc1-Vivek-V6 49,951KiB/s 3,154KiB/s 22,593KiB/s 2,648KiB/s
Performance 0% -2.8% +2.9% -4.4%
============
By libaio calls
Block Size:32K
File Size: 500MMode Normal read | Random read | Normal write | Random write
2.6.31-rc1 49,447KiB/s 3,296KiB/s 57,519KiB/s 21,093KiB/s
2.6.31-rc1-Vivek-V6 50,142KiB/s 3,238KiB/s 57,791KiB/s 21,283KiB/s
Performance +1.4% -1.8% 0% +0.1%
============
_______________________________________________
Containers mailing list
[email protected]
https://lists.linux-foundation.org/mailman/listinfo/containers
Quoted 13 lines · Vivek Goyal
Quoted 5 lines · Gui Jianfeng
Hi,
Here are some fio test results for IO Controller V6 built and not built. Iozone test results are also attached.
Hi Gui,
Thanks a lot for some performance numbers. It seems to be a mixed chart. Performance gains at some places and loss at others. I am curious about that -7.0% for normal writes. Not sure what can contribute to that.
Quoted 6 lines
What was the value of "fairness" parameter when you ran those tests? Can you please set fairness = 0 and re-run the tests (if you have already not done so).
By default fairness is set to 1 in V6. With fairness = 0, we should be very close to existing CFQ behavior. If not, then we need to dive deeper and see why variations are happening.
Hi Vivek,
I tested with default fairness value, i'll re-test it when fairness is set to 0.
Quoted 2 lines
Is it also possible to run the same tests with V7.
Sure.
Signature · 7 lines
Regards Gui Jianfeng _______________________________________________ Containers mailing list [email protected] https://lists.linux-foundation.org/mailman/listinfo/containers
Quoted 13 lines · Vivek Goyal
Quoted 5 lines · Gui Jianfeng
Hi,
Here are some fio test results for IO Controller V6 built and not built. Iozone test results are also attached.
Hi Gui,
Thanks a lot for some performance numbers. It seems to be a mixed chart. Performance gains at some places and loss at others. I am curious about that -7.0% for normal writes. Not sure what can contribute to that.
Quoted 6 lines
What was the value of "fairness" parameter when you ran those tests? Can you please set fairness = 0 and re-run the tests (if you have already not done so).
By default fairness is set to 1 in V6. With fairness = 0, we should be very close to existing CFQ behavior. If not, then we need to dive deeper and see why variations are happening.
Hi Vivek,
I re-run the fio test under V6 for normal reads and writes with fairness == 0. It seems the performance gets a little better for normal writes than before.
Mode Normal read | Random read | Normal write | Random write
2.6.31-rc1 53,547KiB/s 2,894KiB/s 44,088KiB/s 8,450KiB/s
V6(fairness = 0) 53,199KiB/s 2,847KiB/s 41,898KiB/s 8,582KiB/s
Performance 0% -1.6% -4.9% +1.5%
Quoted 73 lines
Is it also possible to run the same tests with V7.
Thanks
VivekQuoted 61 lines
Arch: X86
Mem: 1G
Disk: 320G
IO Scheduler: CFQ============
By normal read and write syscall.Block Size:32K
File Size: 1G * 10Mode Normal read | Random read | Normal write | Random write | Direct read | Direct write
2.6.31-rc1 47,932KiB/s 3,566KiB/s 45,693KiB/s 8,501KiB/s 50,088KiB/s 43,473KiB/s
2.6.31-rc1-Vivek-V6 47,231KiB/s 3,411KiB/s 42,451KiB/s 8,714KiB/s 51,284KiB/s 42,341KiB/s
Performance -1.5% -4.4% -7.0% +2.5% +2.4% -2.6%
============
By mmap.
Block Size:32K
File Size: 500MMode Normal read | Random read | Normal write | Random write
2.6.31-rc1 49,951KiB/s 3,245KiB/s 21,950KiB/s 2,771KiB/s
2.6.31-rc1-Vivek-V6 49,951KiB/s 3,154KiB/s 22,593KiB/s 2,648KiB/s
Performance 0% -2.8% +2.9% -4.4%
============
By libaio calls
Block Size:32K
File Size: 500MMode Normal read | Random read | Normal write | Random write
2.6.31-rc1 49,447KiB/s 3,296KiB/s 57,519KiB/s 21,093KiB/s
2.6.31-rc1-Vivek-V6 50,142KiB/s 3,238KiB/s 57,791KiB/s 21,283KiB/s
Performance +1.4% -1.8% 0% +0.1%
============
Signature · 8 lines
Regards Gui Jianfeng _______________________________________________ Containers mailing list [email protected] https://lists.linux-foundation.org/mailman/listinfo/containers
Quoted 13 lines · Vivek Goyal
Quoted 5 lines · Gui Jianfeng
Hi,
Here are some fio test results for IO Controller V6 built and not built. Iozone test results are also attached.
Hi Gui,
Thanks a lot for some performance numbers. It seems to be a mixed chart. Performance gains at some places and loss at others. I am curious about that -7.0% for normal writes. Not sure what can contribute to that.
Quoted 6 lines
What was the value of "fairness" parameter when you ran those tests? Can you please set fairness = 0 and re-run the tests (if you have already not done so).
By default fairness is set to 1 in V6. With fairness = 0, we should be very close to existing CFQ behavior. If not, then we need to dive deeper and see why variations are happening.
Hi Vivek,
I tested with default fairness value, i'll re-test it when fairness is set to 0.
Quoted 2 lines
Is it also possible to run the same tests with V7.
Sure.
Signature · 2 lines
Regards Gui Jianfeng
Quoted 13 lines · Vivek Goyal
Quoted 5 lines · Gui Jianfeng
Hi,
Here are some fio test results for IO Controller V6 built and not built. Iozone test results are also attached.
Hi Gui,
Thanks a lot for some performance numbers. It seems to be a mixed chart. Performance gains at some places and loss at others. I am curious about that -7.0% for normal writes. Not sure what can contribute to that.
Quoted 6 lines
What was the value of "fairness" parameter when you ran those tests? Can you please set fairness = 0 and re-run the tests (if you have already not done so).
By default fairness is set to 1 in V6. With fairness = 0, we should be very close to existing CFQ behavior. If not, then we need to dive deeper and see why variations are happening.
Hi Vivek,
I re-run the fio test under V6 for normal reads and writes with fairness == 0. It seems the performance gets a little better for normal writes than before.
Mode Normal read | Random read | Normal write | Random write
2.6.31-rc1 53,547KiB/s 2,894KiB/s 44,088KiB/s 8,450KiB/s
V6(fairness = 0) 53,199KiB/s 2,847KiB/s 41,898KiB/s 8,582KiB/s
Performance 0% -1.6% -4.9% +1.5%
Quoted 73 lines
Is it also possible to run the same tests with V7.
Thanks
VivekQuoted 61 lines
Arch: X86
Mem: 1G
Disk: 320G
IO Scheduler: CFQ============
By normal read and write syscall.Block Size:32K
File Size: 1G * 10Mode Normal read | Random read | Normal write | Random write | Direct read | Direct write
2.6.31-rc1 47,932KiB/s 3,566KiB/s 45,693KiB/s 8,501KiB/s 50,088KiB/s 43,473KiB/s
2.6.31-rc1-Vivek-V6 47,231KiB/s 3,411KiB/s 42,451KiB/s 8,714KiB/s 51,284KiB/s 42,341KiB/s
Performance -1.5% -4.4% -7.0% +2.5% +2.4% -2.6%
============
By mmap.
Block Size:32K
File Size: 500MMode Normal read | Random read | Normal write | Random write
2.6.31-rc1 49,951KiB/s 3,245KiB/s 21,950KiB/s 2,771KiB/s
2.6.31-rc1-Vivek-V6 49,951KiB/s 3,154KiB/s 22,593KiB/s 2,648KiB/s
Performance 0% -2.8% +2.9% -4.4%
============
By libaio calls
Block Size:32K
File Size: 500MMode Normal read | Random read | Normal write | Random write
2.6.31-rc1 49,447KiB/s 3,296KiB/s 57,519KiB/s 21,093KiB/s
2.6.31-rc1-Vivek-V6 50,142KiB/s 3,238KiB/s 57,791KiB/s 21,283KiB/s
Performance +1.4% -1.8% 0% +0.1%
============
Signature · 2 lines
Regards Gui Jianfeng
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 5 lines · Gui Jianfeng
Hi,
Here are some fio test results for IO Controller V6 built and not built. Iozone test results are also attached.
Hi Gui,
Thanks a lot for some performance numbers. It seems to be a mixed chart. Performance gains at some places and loss at others. I am curious about that -7.0% for normal writes. Not sure what can contribute to that.
What was the value of "fairness" parameter when you ran those tests? Can you please set fairness = 0 and re-run the tests (if you have already not done so).
By default fairness is set to 1 in V6. With fairness = 0, we should be very close to existing CFQ behavior. If not, then we need to dive deeper and see why variations are happening.
Is it also possible to run the same tests with V7.
Thanks
Vivek
Quoted 43 lines
Arch: X86
Mem: 1G
Disk: 320G
IO Scheduler: CFQ============
By normal read and write syscall.Block Size:32K
File Size: 1G * 10Mode Normal read | Random read | Normal write | Random write | Direct read | Direct write
2.6.31-rc1 47,932KiB/s 3,566KiB/s 45,693KiB/s 8,501KiB/s 50,088KiB/s 43,473KiB/s
2.6.31-rc1-Vivek-V6 47,231KiB/s 3,411KiB/s 42,451KiB/s 8,714KiB/s 51,284KiB/s 42,341KiB/s
Performance -1.5% -4.4% -7.0% +2.5% +2.4% -2.6%
============
By mmap.
Block Size:32K
File Size: 500MMode Normal read | Random read | Normal write | Random write
2.6.31-rc1 49,951KiB/s 3,245KiB/s 21,950KiB/s 2,771KiB/s
2.6.31-rc1-Vivek-V6 49,951KiB/s 3,154KiB/s 22,593KiB/s 2,648KiB/s
Performance 0% -2.8% +2.9% -4.4%
============
By libaio calls
Block Size:32K
File Size: 500MMode Normal read | Random read | Normal write | Random write
Quoted 18 lines
2.6.31-rc1 49,447KiB/s 3,296KiB/s 57,519KiB/s 21,093KiB/s
2.6.31-rc1-Vivek-V6 50,142KiB/s 3,238KiB/s 57,791KiB/s 21,283KiB/s
Performance +1.4% -1.8% 0% +0.1%
============
_______________________________________________
Containers mailing list
[email protected]
https://lists.linux-foundation.org/mailman/listinfo/containers