cfq-iosced: Implement IOPS mode and group_idle tunable V3

o Implement another CFQ mode where we charge queue/group in terms of number of requests dispatched instead of measuring the time. Measuring in terms of time is not possible when we are driving deeper queue depths and there are requests from multiple cfq queues in the request queue.

o This mode currently gets activated if one sets slice_idle=0 and associated disk supports NCQ. Again the idea is that on an NCQ disk with idling disabled most of the queues will dispatch 1 or more requests and then cfq queue expiry happens and we don't have a way to measure time. So start providing fairness in terms of IOPS.

o Currently this primarily is beneficial with cfq group scheduling where one can disable slice idling so that we don't idle on queue and drive deeper request queue deptsh (achieving better throughput), at the same time group idle is enabled so one should get service differentiation among groups.

Quoted 11 lines · Vivek Goyal

o Implement another CFQ mode where we charge queue/group in terms of number of requests dispatched instead of measuring the time. Measuring in terms of time is not possible when we are driving deeper queue depths and there are requests from multiple cfq queues in the request queue.

o This mode currently gets activated if one sets slice_idle=0 and associated disk supports NCQ. Again the idea is that on an NCQ disk with idling disabled most of the queues will dispatch 1 or more requests and then cfq queue expiry happens and we don't have a way to measure time. So start providing fairness in terms of IOPS.

Quoted 4 lines

o Currently this primarily is beneficial with cfq group scheduling where one can disable slice idling so that we don't idle on queue and drive deeper request queue deptsh (achieving better throughput), at the same time group idle is enabled so one should get service differentiation among groups.

I like that this is more isolated now. I'm slowly warming up to it. I have one question--just a curiosity, really. What do you see now for the reported sl_used in blktrace when slice_idle is zero and the hardware supports command queueing?

Cheers,
Jeff

o Implement a new tunable group_idle, which allows idling on the group instead of a cfq queue. Hence one can set slice_idle = 0 and not idle on the individual queues but idle on the group. This way on fast storage we can get fairness between groups at the same time overall throughput improves.

Signed-off-by Vivek Goyal <[email protected]>

---

block/cfq-iosched.c +1 -0

@@ -1929,13 +1941,21 @@ static void cfq_arm_slice_timer(struct cfq_data *cfqd)
 		return;
 	}
 
+	/* There are other queues in the group, don't do group idle */

Quoted 8 lines · Vivek Goyal

o Implement a new tunable group_idle, which allows idling on the group instead of a cfq queue. Hence one can set slice_idle = 0 and not idle on the individual queues but idle on the group. This way on fast storage we can get fairness between groups at the same time overall throughput improves.

Signed-off-by Vivek Goyal <[email protected]>

---

[snip]

Quoted 10 lines

block/cfq-iosched.c +4 -0

@@ -1929,13 +1941,21 @@ static void cfq_arm_slice_timer(struct cfq_data *cfqd)
 		return;
 	}
 
+	/* There are other queues in the group, don't do group idle */
+	if (group_idle && cfqq->cfqg->nr_cfqq > 1)
+		return;
+
 	cfq_mark_cfqq_wait_request(cfqq);
 

block/cfq-iosched.c static void cfq_arm_slice_timer(struct cfq_data *cfqd) · line 1953

Quoted 5 lines

block/cfq-iosched.c +4 -1

-	sl = cfqd->cfq_slice_idle;
+	if (group_idle)
+		sl = cfqd->cfq_group_idle;
+	else
+		sl = cfqd->cfq_slice_idle;

What happens when both group_idle and slice_idle are set?

Is that a
sane thing to do from a user's perspective?

If not, please protect
against it in the configuration code. If so, then explain why we prefer group_idle here, but slice_idle in completed request for the extend_sl:

block/cfq-iosched.c static void cfq_completed_request(struct request_queue *q, struct request *rq) · line 3464

Quoted 9 lines

block/cfq-iosched.c +4 -1

@@ -3425,7 +3458,10 @@ static void cfq_completed_request(struct request_queue *q, struct request *rq)
 		 * the queue.
 		 */
 		if (cfq_should_wait_busy(cfqd, cfqq)) {
-			cfqq->slice_end = jiffies + cfqd->cfq_slice_idle;
+			unsigned long extend_sl = cfqd->cfq_slice_idle;
+			if (!cfqd->cfq_slice_idle)
+				extend_sl = cfqd->cfq_group_idle;
+			cfqq->slice_end = jiffies + extend_sl;

Also, you'll need to add documentation for this new tunable.

Cheers,
Jeff

Quoted 21 lines · Jeff Moyer

Quoted 8 lines · Vivek Goyal

o Implement a new tunable group_idle, which allows idling on the group instead of a cfq queue. Hence one can set slice_idle = 0 and not idle on the individual queues but idle on the group. This way on fast storage we can get fairness between groups at the same time overall throughput improves.

Signed-off-by Vivek Goyal <[email protected]>

---

[snip]

Quoted 10 lines

block/cfq-iosched.c +4 -0

@@ -1929,13 +1941,21 @@ static void cfq_arm_slice_timer(struct cfq_data *cfqd)
 		return;
 	}
 
+	/* There are other queues in the group, don't do group idle */
+	if (group_idle && cfqq->cfqg->nr_cfqq > 1)
+		return;
+
 	cfq_mark_cfqq_wait_request(cfqq);
 

Quoted 7 lines

block/cfq-iosched.c static void cfq_arm_slice_timer(struct cfq_data *cfqd) · line 1953

Quoted 5 lines

block/cfq-iosched.c +4 -1

-	sl = cfqd->cfq_slice_idle;
+	if (group_idle)
+		sl = cfqd->cfq_group_idle;
+	else
+		sl = cfqd->cfq_slice_idle;

What happens when both group_idle and slice_idle are set?

slice_idle prevails. Notice that "group_idle" is a local variable which is set to 1 only if we decide not to idle on the cfq queue.

On high end storage (I got on HP EVA storage array with 12 SATA disks in RAID 5),

So this is not the default mode. This new tunable group_idle, allows one to set slice_idle=0 to disable some of the CFQ features and and use primarily group service differentation feature.

Quoted 2 lines · Vivek Goyal

On high end storage (I got on HP EVA storage array with 12 SATA disks in RAID 5),

That's actually quite low end storage for a server these days :)

Quoted 3 lines

So this is not the default mode. This new tunable group_idle, allows one to set slice_idle=0 to disable some of the CFQ features and and use primarily group service differentation feature.

While this is better than before needing a sysfs tweak to get any performance out of any kind of server class hardware still is pretty horrible. And slice_idle=0 is not exactly the most obvious paramter
I would look for either. So having some way to automatically disable
this mode based on hardware characteristics would be really useful,

and if that's not possible at least make sure it's very obviously document and easily found using web searches.

Btw, what effect does slice_idle=0 with your patches have to single SATA disk and single SSD setups?

To me this sounds like slice_idle=0 is the right default then, as it gives useful behaviour for all systems linux runs on.

 Setups with
more than a few spindles are for sure more common than setups making use of cgroups.  Especially given that cgroups are more of a high end feature you'd rarely use on a single SATA spindle anyway.  So setting a paramter to make this useful sounds like the much better option.

Especially given that the block cgroup code doesn't work particularly well in presence of barriers, which are on for any kind of real life production setup anyway.

Quoted 2 lines · Christoph Hellwig

To me this sounds like slice_idle=0 is the right default then, as it gives useful behaviour for all systems linux runs on.

No, it will give bad performance on single disks, possibly worse than deadline (deadline at least sorts the requests between different queues, while CFQ with slice_idle=0 doesn't even do this for readers).

Setting slice_idle to 0 should be considered only when a single sequential reader cannot saturate the disk bandwidth, and this happens only on smart enough hardware with large number of spindles.

Quoted 10 lines · Corrado Zoccolo

Quoted 2 lines · Christoph Hellwig

To me this sounds like slice_idle=0 is the right default then, as it gives useful behaviour for all systems linux runs on.

No, it will give bad performance on single disks, possibly worse than deadline (deadline at least sorts the requests between different queues, while CFQ with slice_idle=0 doesn't even do this for readers).

Setting slice_idle to 0 should be considered only when a single sequential reader cannot saturate the disk bandwidth, and this happens only on smart enough hardware with large number of spindles.

I was thinking of writting a user space utility which can launch increasing number of parallel direct/buffered reads from device and if device can sustain more than 1 parallel reads with increasing throughput, then it probably is good indicator that one might be better off with slice_idle=0.

Will try that today...

Quoted 2 lines · Christoph Hellwig

To me this sounds like slice_idle=0 is the right default then, as it gives useful behaviour for all systems linux runs on.

No, it will give bad performance on single disks, possibly worse than deadline (deadline at least sorts the requests between different queues, while CFQ with slice_idle=0 doesn't even do this for readers).

Quoted 6 lines · Corrado Zoccolo

Quoted 2 lines · Christoph Hellwig

To me this sounds like slice_idle=0 is the right default then, as it gives useful behaviour for all systems linux runs on.

No, it will give bad performance on single disks, possibly worse than deadline (deadline at least sorts the requests between different queues, while CFQ with slice_idle=0 doesn't even do this for readers).

Not sure if CFQ will be worse than deadline with slice_idle=0. CFQ has some inbuilt things which should help.

- Readers preempt Writers
- All writers go in one single queue (at one prio level), readers get their individual queues and can outnumber writers.

So I guess CFQ with slice_idle=0 should not be worse than deadline in terms of read latencies.

Vivek

To me this sounds like slice_idle=0 is the right default then, as it gives useful behaviour for all systems linux runs on. Setups with more than a few spindles are for sure more common than setups making use of cgroups. Especially given that cgroups are more of a high end feature you'd rarely use on a single SATA spindle anyway. So setting a paramter to make this useful sounds like the much better option.

Especially given that the block cgroup code doesn't work particularly well in presence of barriers, which are on for any kind of real life production setup anyway.

Quoted 2 lines · Vivek Goyal

On high end storage (I got on HP EVA storage array with 12 SATA disks in RAID 5),

That's actually quite low end storage for a server these days :)

Quoted 3 lines

So this is not the default mode. This new tunable group_idle, allows one to set slice_idle=0 to disable some of the CFQ features and and use primarily group service differentation feature.

While this is better than before needing a sysfs tweak to get any performance out of any kind of server class hardware still is pretty horrible. And slice_idle=0 is not exactly the most obvious paramter
I would look for either. So having some way to automatically disable
this mode based on hardware characteristics would be really useful, and if that's not possible at least make sure it's very obviously document and easily found using web searches.

Btw, what effect does slice_idle=0 with your patches have to single SATA disk and single SSD setups?

Hi,

This is V3 of the group_idle and CFQ IOPS mode implementation patchset. Since V2 I have cleaned up the code a bit to clarify the confusion lingering around in what cases do we charge time slice and in what cases do we charge number of requests.

What's the problem
------------------
On high end storage (I got on HP EVA storage array with 12 SATA disks in RAID 5), CFQ's model of dispatching requests from a single queue at a time (sequential readers/write sync writers etc), becomes a bottleneck. Often we don't drive enough request queue depth to keep all the disks busy and suffer a lot in terms of overall throughput.

All these problems primarily originate from two things. Idling on per cfq queue and quantum (dispatching limited number of requests from a single queue) and till then not allowing dispatch from other queues. Once you set the slice_idle=0 and quantum to higher value, most of the CFQ's problem on higher end storage disappear.

This problem also becomes visible in IO controller where one creates multiple groups and gets the fairness but overall throughput is less. In the following table, I am running increasing number of sequential readers (1,2,4,8) in 8 groups of weight 100 to 800.

Kernel=2.6.35-rc5-iops+
GROUPMODE=1 NRGRP=8
DIR=/mnt/iostestmnt/fio DEV=/dev/dm-4
Workload=bsr iosched=cfq Filesz=512M bs=4K
group_isolation=1 slice_idle=8 group_idle=8 quantum=8
=========================================================================
AVERAGE[bsr] [bw in KB/s]
-------

job     Set NR  cgrp1  cgrp2  cgrp3  cgrp4  cgrp5  cgrp6  cgrp7  cgrp8  total
---     --- --  ---------------------------------------------------------------
bsr     3   1   6186   12752  16568  23068  28608  35785  42322  48409  213701
bsr     3   2   5396   10902  16959  23471  25099  30643  37168  42820  192461
bsr     3   4   4655   9463   14042  20537  24074  28499  34679  37895  173847
bsr     3   8   4418   8783   12625  19015  21933  26354  29830  36290  159249

Notice that overall throughput is just around 160MB/s with 8 sequential reader in each group.

With this patch set, I have set slice_idle=0 and re-ran same test.

Kernel=2.6.35-rc5-iops+
GROUPMODE=1 NRGRP=8
DIR=/mnt/iostestmnt/fio DEV=/dev/dm-4
Workload=bsr iosched=cfq Filesz=512M bs=4K
group_isolation=1 slice_idle=0 group_idle=8 quantum=8
=========================================================================
AVERAGE[bsr] [bw in KB/s]
-------

job     Set NR  cgrp1  cgrp2  cgrp3  cgrp4  cgrp5  cgrp6  cgrp7  cgrp8  total
---     --- --  ---------------------------------------------------------------
bsr     3   1   6523   12399  18116  24752  30481  36144  42185  48894  219496
bsr     3   2   10072  20078  29614  38378  46354  52513  58315  64833  320159
bsr     3   4   11045  22340  33013  44330  52663  58254  63883  70990  356520
bsr     3   8   12362  25860  37920  47486  61415  47292  45581  70828  348747

Notice how overall throughput has shot upto 348MB/s while retaining the ability to do the IO control.

So this is not the default mode. This new tunable group_idle, allows one to set slice_idle=0 to disable some of the CFQ features and and use primarily group service differentation feature.

If you have thoughts on other ways of solving the problem, I am all ears to it.

Thanks
Vivek

Quoted 68 lines · Vivek Goyal

Hi,

This is V3 of the group_idle and CFQ IOPS mode implementation patchset. Since V2 I have cleaned up the code a bit to clarify the confusion lingering around in what cases do we charge time slice and in what cases do we charge number of requests.

What's the problem
------------------
On high end storage (I got on HP EVA storage array with 12 SATA disks in RAID 5), CFQ's model of dispatching requests from a single queue at a time (sequential readers/write sync writers etc), becomes a bottleneck. Often we don't drive enough request queue depth to keep all the disks busy and suffer a lot in terms of overall throughput.

All these problems primarily originate from two things. Idling on per cfq queue and quantum (dispatching limited number of requests from a single queue) and till then not allowing dispatch from other queues. Once you set the slice_idle=0 and quantum to higher value, most of the CFQ's problem on higher end storage disappear.

This problem also becomes visible in IO controller where one creates multiple groups and gets the fairness but overall throughput is less. In the following table, I am running increasing number of sequential readers (1,2,4,8) in 8 groups of weight 100 to 800.

Kernel=2.6.35-rc5-iops+
GROUPMODE=1 NRGRP=8
DIR=/mnt/iostestmnt/fio DEV=/dev/dm-4
Workload=bsr iosched=cfq Filesz=512M bs=4K
group_isolation=1 slice_idle=8 group_idle=8 quantum=8
=========================================================================
AVERAGE[bsr] [bw in KB/s]
-------

job     Set NR  cgrp1  cgrp2  cgrp3  cgrp4  cgrp5  cgrp6  cgrp7  cgrp8  total
---     --- --  ---------------------------------------------------------------
bsr     3   1   6186   12752  16568  23068  28608  35785  42322  48409  213701
bsr     3   2   5396   10902  16959  23471  25099  30643  37168  42820  192461
bsr     3   4   4655   9463   14042  20537  24074  28499  34679  37895  173847
bsr     3   8   4418   8783   12625  19015  21933  26354  29830  36290  159249

Notice that overall throughput is just around 160MB/s with 8 sequential reader in each group.

With this patch set, I have set slice_idle=0 and re-ran same test.

Kernel=2.6.35-rc5-iops+
GROUPMODE=1 NRGRP=8
DIR=/mnt/iostestmnt/fio DEV=/dev/dm-4
Workload=bsr iosched=cfq Filesz=512M bs=4K
group_isolation=1 slice_idle=0 group_idle=8 quantum=8
=========================================================================
AVERAGE[bsr] [bw in KB/s]
-------

job     Set NR  cgrp1  cgrp2  cgrp3  cgrp4  cgrp5  cgrp6  cgrp7  cgrp8  total
---     --- --  ---------------------------------------------------------------
bsr     3   1   6523   12399  18116  24752  30481  36144  42185  48894  219496
bsr     3   2   10072  20078  29614  38378  46354  52513  58315  64833  320159
bsr     3   4   11045  22340  33013  44330  52663  58254  63883  70990  356520
bsr     3   8   12362  25860  37920  47486  61415  47292  45581  70828  348747

Notice how overall throughput has shot upto 348MB/s while retaining the ability to do the IO control.

So this is not the default mode. This new tunable group_idle, allows one to set slice_idle=0 to disable some of the CFQ features and and use primarily group service differentation feature.

Quoted 2 lines

If you have thoughts on other ways of solving the problem, I am all ears to it.

Hi Vivek

Would you attach your fio job config file?

Quoted 69 lines · Gui Jianfeng

Quoted 68 lines · Vivek Goyal

Hi,

This is V3 of the group_idle and CFQ IOPS mode implementation patchset. Since V2 I have cleaned up the code a bit to clarify the confusion lingering around in what cases do we charge time slice and in what cases do we charge number of requests.

What's the problem
------------------
On high end storage (I got on HP EVA storage array with 12 SATA disks in RAID 5), CFQ's model of dispatching requests from a single queue at a time (sequential readers/write sync writers etc), becomes a bottleneck. Often we don't drive enough request queue depth to keep all the disks busy and suffer a lot in terms of overall throughput.

All these problems primarily originate from two things. Idling on per cfq queue and quantum (dispatching limited number of requests from a single queue) and till then not allowing dispatch from other queues. Once you set the slice_idle=0 and quantum to higher value, most of the CFQ's problem on higher end storage disappear.

This problem also becomes visible in IO controller where one creates multiple groups and gets the fairness but overall throughput is less. In the following table, I am running increasing number of sequential readers (1,2,4,8) in 8 groups of weight 100 to 800.

Kernel=2.6.35-rc5-iops+
GROUPMODE=1 NRGRP=8
DIR=/mnt/iostestmnt/fio DEV=/dev/dm-4
Workload=bsr iosched=cfq Filesz=512M bs=4K
group_isolation=1 slice_idle=8 group_idle=8 quantum=8
=========================================================================
AVERAGE[bsr] [bw in KB/s]
-------

job     Set NR  cgrp1  cgrp2  cgrp3  cgrp4  cgrp5  cgrp6  cgrp7  cgrp8  total
---     --- --  ---------------------------------------------------------------
bsr     3   1   6186   12752  16568  23068  28608  35785  42322  48409  213701
bsr     3   2   5396   10902  16959  23471  25099  30643  37168  42820  192461
bsr     3   4   4655   9463   14042  20537  24074  28499  34679  37895  173847
bsr     3   8   4418   8783   12625  19015  21933  26354  29830  36290  159249

Notice that overall throughput is just around 160MB/s with 8 sequential reader in each group.

With this patch set, I have set slice_idle=0 and re-ran same test.

Kernel=2.6.35-rc5-iops+
GROUPMODE=1 NRGRP=8
DIR=/mnt/iostestmnt/fio DEV=/dev/dm-4
Workload=bsr iosched=cfq Filesz=512M bs=4K
group_isolation=1 slice_idle=0 group_idle=8 quantum=8
=========================================================================
AVERAGE[bsr] [bw in KB/s]
-------

job     Set NR  cgrp1  cgrp2  cgrp3  cgrp4  cgrp5  cgrp6  cgrp7  cgrp8  total
---     --- --  ---------------------------------------------------------------
bsr     3   1   6523   12399  18116  24752  30481  36144  42185  48894  219496
bsr     3   2   10072  20078  29614  38378  46354  52513  58315  64833  320159
bsr     3   4   11045  22340  33013  44330  52663  58254  63883  70990  356520
bsr     3   8   12362  25860  37920  47486  61415  47292  45581  70828  348747

Notice how overall throughput has shot upto 348MB/s while retaining the ability to do the IO control.

So this is not the default mode. This new tunable group_idle, allows one to set slice_idle=0 to disable some of the CFQ features and and use primarily group service differentation feature.

Quoted 6 lines

Quoted 2 lines

If you have thoughts on other ways of solving the problem, I am all ears to it.

Hi Vivek

Would you attach your fio job config file?

Hi Gui,

I have written a fio based test script, "iostest", to be able to do cgroup and other IO scheduler testing more smoothly and I am using that. I am attaching the compressed script with the mail. Try using it and if it works for you and you find it useful, I can think of hosting a git tree somewhere.

I used following following command lines to test above.

# iostest <block-device> -G -w bsr -m 8 -c --nrgrp 8 --total

With slice idle disabled.

# iostest <block-device> -G -w bsr -m 8 -c --nrgrp 8 --total -I 0

Thanks
Vivek

Hi,

This is V3 of the group_idle and CFQ IOPS mode implementation patchset. Since V2 I have cleaned up the code a bit to clarify the confusion lingering around in what cases do we charge time slice and in what cases do we charge number of requests.

What's the problem
------------------
On high end storage (I got on HP EVA storage array with 12 SATA disks in RAID 5), CFQ's model of dispatching requests from a single queue at a time (sequential readers/write sync writers etc), becomes a bottleneck. Often we don't drive enough request queue depth to keep all the disks busy and suffer a lot in terms of overall throughput.

All these problems primarily originate from two things. Idling on per cfq queue and quantum (dispatching limited number of requests from a single queue) and till then not allowing dispatch from other queues. Once you set the slice_idle=0 and quantum to higher value, most of the CFQ's problem on higher end storage disappear.

This problem also becomes visible in IO controller where one creates multiple groups and gets the fairness but overall throughput is less. In the following table, I am running increasing number of sequential readers (1,2,4,8) in 8 groups of weight 100 to 800.

Kernel=2.6.35-rc5-iops+
GROUPMODE=1 NRGRP=8
DIR=/mnt/iostestmnt/fio DEV=/dev/dm-4
Workload=bsr iosched=cfq Filesz=512M bs=4K
group_isolation=1 slice_idle=8 group_idle=8 quantum=8
=========================================================================
AVERAGE[bsr] [bw in KB/s]
-------

job     Set NR  cgrp1  cgrp2  cgrp3  cgrp4  cgrp5  cgrp6  cgrp7  cgrp8  total
---     --- --  ---------------------------------------------------------------
bsr     3   1   6186   12752  16568  23068  28608  35785  42322  48409  213701
bsr     3   2   5396   10902  16959  23471  25099  30643  37168  42820  192461
bsr     3   4   4655   9463   14042  20537  24074  28499  34679  37895  173847
bsr     3   8   4418   8783   12625  19015  21933  26354  29830  36290  159249

Notice that overall throughput is just around 160MB/s with 8 sequential reader in each group.

With this patch set, I have set slice_idle=0 and re-ran same test.

Kernel=2.6.35-rc5-iops+
GROUPMODE=1 NRGRP=8
DIR=/mnt/iostestmnt/fio DEV=/dev/dm-4
Workload=bsr iosched=cfq Filesz=512M bs=4K
group_isolation=1 slice_idle=0 group_idle=8 quantum=8
=========================================================================
AVERAGE[bsr] [bw in KB/s]
-------

job     Set NR  cgrp1  cgrp2  cgrp3  cgrp4  cgrp5  cgrp6  cgrp7  cgrp8  total
---     --- --  ---------------------------------------------------------------
bsr     3   1   6523   12399  18116  24752  30481  36144  42185  48894  219496
bsr     3   2   10072  20078  29614  38378  46354  52513  58315  64833  320159
bsr     3   4   11045  22340  33013  44330  52663  58254  63883  70990  356520
bsr     3   8   12362  25860  37920  47486  61415  47292  45581  70828  348747

Notice how overall throughput has shot upto 348MB/s while retaining the ability to do the IO control.

So this is not the default mode. This new tunable group_idle, allows one to set slice_idle=0 to disable some of the CFQ features and and use primarily group service differentation feature.

If you have thoughts on other ways of solving the problem, I am all ears to it.

Thanks
Vivek

Hi Vivek,

I did some tests on single SATA disk on my desktop. With patches applied, seems no regression occurs till now, and have some performance improvement in case of "Direct Random Reader" mode. Here're some numbers on my box.

Vallina kernel:

Blkio is already mounted at /cgroup/blkio. Unmounting it
DIR=/mnt/iostestmnt/fio DEV=/dev/sdb2
GROUPMODE=1 NRGRP=4
Will run workloads for increasing number of threads upto a max of 4 Starting test for [drr] with set=1 numjobs=1 filesz=512M bs=32k runtime=30 Starting test for [drr] with set=1 numjobs=2 filesz=512M bs=32k runtime=30 Starting test for [drr] with set=1 numjobs=4 filesz=512M bs=32k runtime=30 Finished test for workload [drr]
Host=localhost.localdomain Kernel=2.6.35-rc4-Vivek-+
GROUPMODE=1 NRGRP=4
DIR=/mnt/iostestmnt/fio DEV=/dev/sdb2
Workload=drr iosched=cfq Filesz=512M bs=32k
group_isolation=1 slice_idle=0 group_idle=8 quantum=8
=========================================================================
AVERAGE[drr] [bw in KB/s]
-------

job     Set NR  cgrp1  cgrp2  cgrp3  cgrp4  total
---     --- --  -----------------------------------
drr     1   1   761    761    762    760    3044
drr     1   2   185    420    727    1256   2588
drr     1   4   180    371    588    863    2002

Patched kernel:

Blkio is already mounted at /cgroup/blkio. Unmounting it
DIR=/mnt/iostestmnt/fio DEV=/dev/sdb2
GROUPMODE=1 NRGRP=4
Will run workloads for increasing number of threads upto a max of 4 Starting test for [drr] with set=1 numjobs=1 filesz=512M bs=32k runtime=30 Starting test for [drr] with set=1 numjobs=2 filesz=512M bs=32k runtime=30 Starting test for [drr] with set=1 numjobs=4 filesz=512M bs=32k runtime=30 Finished test for workload [drr]
Host=localhost.localdomain Kernel=2.6.35-rc4-Vivek-+
GROUPMODE=1 NRGRP=4
DIR=/mnt/iostestmnt/fio DEV=/dev/sdb2
Workload=drr iosched=cfq Filesz=512M bs=32k
group_isolation=1 slice_idle=0 group_idle=8 quantum=8
=========================================================================
AVERAGE[drr] [bw in KB/s]
-------

job     Set NR  cgrp1  cgrp2  cgrp3  cgrp4  total
---     --- --  -----------------------------------
drr     1   1   323    671    1030   1378   3402
drr     1   2   165    391    686    1144   2386
drr     1   4   185    373    612    873    2043

Thanks
Gui

Quoted 4 lines

Thanks
Vivek

On Mon, Jul 26, 2010 at 02:58:16PM +0800, Gui Jianfeng wrote:

[..]

Quoted 2 lines

Hi Vivek,

Quoted 4 lines

I did some tests on single SATA disk on my desktop. With patches applied, seems no regression occurs till now, and have some performance improvement in case of "Direct Random Reader" mode. Here're some numbers on my box.

Thanks for testing Gui. "iostest" seems to be working for you. If you had to some fixes to make it work on my boxes, do send those to me, and I can commit those in my internal git tree.

After running the script, you can also run "iostest -R <result-dir>" and that will generate a report. It will not have all this "Starting test..." lines and looks nicer.

Good to know that you don't see any regressions on SATA disk in your cgroup testing with this patchset. Little improvement in "drr" might be due to the fact that with existing slice_idle=0, we can still do some extra idling on service tree and first patch in the series (V4) gets rid of that.

Thanks
Vivek

Quoted 55 lines

Vallina kernel:

Blkio is already mounted at /cgroup/blkio. Unmounting it
DIR=/mnt/iostestmnt/fio DEV=/dev/sdb2
GROUPMODE=1 NRGRP=4
Will run workloads for increasing number of threads upto a max of 4 Starting test for [drr] with set=1 numjobs=1 filesz=512M bs=32k runtime=30 Starting test for [drr] with set=1 numjobs=2 filesz=512M bs=32k runtime=30 Starting test for [drr] with set=1 numjobs=4 filesz=512M bs=32k runtime=30 Finished test for workload [drr]
Host=localhost.localdomain Kernel=2.6.35-rc4-Vivek-+
GROUPMODE=1 NRGRP=4
DIR=/mnt/iostestmnt/fio DEV=/dev/sdb2
Workload=drr iosched=cfq Filesz=512M bs=32k
group_isolation=1 slice_idle=0 group_idle=8 quantum=8
=========================================================================
AVERAGE[drr] [bw in KB/s]
-------

job     Set NR  cgrp1  cgrp2  cgrp3  cgrp4  total
---     --- --  -----------------------------------
drr     1   1   761    761    762    760    3044
drr     1   2   185    420    727    1256   2588
drr     1   4   180    371    588    863    2002

Patched kernel:

Blkio is already mounted at /cgroup/blkio. Unmounting it
DIR=/mnt/iostestmnt/fio DEV=/dev/sdb2
GROUPMODE=1 NRGRP=4
Will run workloads for increasing number of threads upto a max of 4 Starting test for [drr] with set=1 numjobs=1 filesz=512M bs=32k runtime=30 Starting test for [drr] with set=1 numjobs=2 filesz=512M bs=32k runtime=30 Starting test for [drr] with set=1 numjobs=4 filesz=512M bs=32k runtime=30 Finished test for workload [drr]
Host=localhost.localdomain Kernel=2.6.35-rc4-Vivek-+
GROUPMODE=1 NRGRP=4
DIR=/mnt/iostestmnt/fio DEV=/dev/sdb2
Workload=drr iosched=cfq Filesz=512M bs=32k
group_isolation=1 slice_idle=0 group_idle=8 quantum=8
=========================================================================
AVERAGE[drr] [bw in KB/s]
-------

job     Set NR  cgrp1  cgrp2  cgrp3  cgrp4  total
---     --- --  -----------------------------------
drr     1   1   323    671    1030   1378   3402
drr     1   2   165    391    686    1144   2386
drr     1   4   185    373    612    873    2043

Thanks
Gui

Quoted 3 lines

Thanks
Vivek