Introduce CFQ group hierarchical scheduling and "use_hierarchy" interface

PATCH Superseded Gui Jianfeng 1 participant 1 message Dec 27, 2010

Share & tools

Gui Jianfeng

author

2010-12-27 08:50 UTC #0 lore raw

v3 cover Introduce CFQ group hierarchical scheduling and "use_hierarchy" interface

Previously, I posted a patchset to add support of CFQ group hierarchical scheduling in the way that it puts all CFQ queues in a hidden group and schedules with other CFQ group under their parent. The patchset is available here,
http://lkml.org/lkml/2010/8/30/30

Vivek think this approach isn't so instinct that we should treat CFQ queues and groups at the same level. Here is the new approach for hierarchical scheduling based on Vivek's suggestion. The most big change of CFQ is that it gets rid of cfq_slice_offset logic, and makes use of vdisktime for CFQ queue scheduling just like CFQ group does. But I still give cfqq some jump in vdisktime based on ioprio, thanks for Vivek to point out this. Now CFQ queue and CFQ group use the same scheduling algorithm.

"use_hierarchy" interface is now added to switch between hierarchical mode and flat mode. It works as memcg.

--
V2 -> V3 Changes:
- Starting from cfqd->grp_service_tree for both hierarchical mode and flat mode
- Avoid recursion when allocating cfqg and force dispatch logic
- Fix a bug when boosting vdisktime
- Adjusting total_weight accordingly when changing weight
- Change group slice calculation into a hierarchical way
- Keep flat mode rather than deleting it first then adding it later
- kfree the parent cfqg if there nobody references to it
- Simplify select_queue logic by using some wrap function
- Make "use_hierarchy" interface work as memcg
- Make use of time_before() for vdisktime compare
- Update Document
- Fix some code style problems

--
V1 -> V2 Changes:
- Raname "struct io_sched_entity" to "struct cfq_entity" and don't differentiate queue_entity and group_entity, just use cfqe instead.
- Give newly added cfqq a small vdisktime jump accord to its ioprio.
- Make flat mode as default CFQ group scheduling mode.
- Introduce "use_hierarchy" interface.
- Update blkio cgroup documents

Benchmarks:
I made use of Vivek's iostest to perform some benchmarks on my box. I tested different workloads with idling on and off. I didn't see any performance drop comparing to vanilla kernel. The attached files are some performance numbers on vanilla Kernel and patched Kernel.

[PATCH 1/6 v3] cfq-iosched: Introduce cfq_entity for CFQ queue
[PATCH 2/6 v3] cfq-iosched: Introduce cfq_entity for CFQ group
[PATCH 3/6 v3] cfq-iosched: Introduce vdisktime and io weight for CFQ queue
[PATCH 4/6 v3] cfq-iosched: Extract some common code of service tree handling for CFQ queue and CFQ group [PATCH 5/6 v3] cfq-iosched: CFQ group hierarchical scheduling and use_hierarchy interface [PATCH 6/6 v3] blkio-cgroup: Document for blkio.use_hierarchy interface

Host=localhost.localdomain Kernel=2.6.37-rc2-Block-+
GROUPMODE=1 NRGRP=4
DIR=/mnt/iostestmnt/fio DEV=/dev/sdb2
Workload=brr iosched=cfq Filesz=512M bs=32k
group_isolation=1 slice_idle=0 group_idle=0 quantum=8
=========================================================================
AVERAGE[brr] [bw in KB/s]
-------

job     Set NR  cgrp1  cgrp2  cgrp3  cgrp4  total  
---     --- --  -----------------------------------
brr     1   1   748    748    749    747    2992   
brr     1   2   256    566    865    896    2583   
brr     1   4   154    318    560    1084   2116

Host=localhost.localdomain Kernel=2.6.37-rc2-Block-+
GROUPMODE=1 NRGRP=4
DIR=/mnt/iostestmnt/fio DEV=/dev/sdb2
Workload=brr iosched=cfq Filesz=512M bs=32k
group_isolation=1 slice_idle=8 group_idle=8 quantum=8
=========================================================================
AVERAGE[brr] [bw in KB/s]
-------

job     Set NR  cgrp1  cgrp2  cgrp3  cgrp4  total  
---     --- --  -----------------------------------
brr     1   1   380    728    1074   1443   3625   
brr     1   2   201    427    713    1252   2593   
brr     1   4   146    327    551    1111   2135

Host=localhost.localdomain Kernel=2.6.37-rc2-Block-+
GROUPMODE=1 NRGRP=4
DIR=/mnt/iostestmnt/fio DEV=/dev/sdb2
Workload=bsr iosched=cfq Filesz=512M bs=32k
group_isolation=1 slice_idle=0 group_idle=0 quantum=8
=========================================================================
AVERAGE[bsr] [bw in KB/s]
-------

job     Set NR  cgrp1  cgrp2  cgrp3  cgrp4  total  
---     --- --  -----------------------------------
bsr     1   1   7797   7799   7788   7805   31189  
bsr     1   2   3026   6433   9643   9953   29055  
bsr     1   4   2302   4582   7507   13557  27948

Host=localhost.localdomain Kernel=2.6.37-rc2-Block-+
GROUPMODE=1 NRGRP=4
DIR=/mnt/iostestmnt/fio DEV=/dev/sdb2
Workload=bsr iosched=cfq Filesz=512M bs=32k
group_isolation=1 slice_idle=8 group_idle=8 quantum=8
=========================================================================
AVERAGE[bsr] [bw in KB/s]
-------

job     Set NR  cgrp1  cgrp2  cgrp3  cgrp4  total  
---     --- --  -----------------------------------
bsr     1   1   6288   10419  15332  20832  52871

Host=localhost.localdomain Kernel=2.6.37-rc2-Block-+
GROUPMODE=1 NRGRP=4
DIR=/mnt/iostestmnt/fio DEV=/dev/sdb2
Workload=drr iosched=cfq Filesz=512M bs=32k
group_isolation=1 slice_idle=0 group_idle=0 quantum=8
=========================================================================
AVERAGE[drr] [bw in KB/s]
-------

job     Set NR  cgrp1  cgrp2  cgrp3  cgrp4  total  
---     --- --  -----------------------------------
drr     1   1   757    757    757    755    3026   
drr     1   2   233    547    829    919    2528   
drr     1   4   143    306    574    1071   2094

Host=localhost.localdomain Kernel=2.6.37-rc2-Block-+
GROUPMODE=1 NRGRP=4
DIR=/mnt/iostestmnt/fio DEV=/dev/sdb2
Workload=drr iosched=cfq Filesz=512M bs=32k
group_isolation=1 slice_idle=8 group_idle=8 quantum=8
=========================================================================
AVERAGE[drr] [bw in KB/s]
-------

job     Set NR  cgrp1  cgrp2  cgrp3  cgrp4  total  
---     --- --  -----------------------------------
drr     1   1   371    729    1080   1429   3609   
drr     1   2   192    409    793    1260   2654   
drr     1   4   153    316    578    1066   2113

Host=localhost.localdomain Kernel=2.6.37-rc2-Block-+
GROUPMODE=1 NRGRP=4
DIR=/mnt/iostestmnt/fio DEV=/dev/sdb2
Workload=drw iosched=cfq Filesz=512M bs=32k
group_isolation=1 slice_idle=0 group_idle=0 quantum=8
=========================================================================
AVERAGE[drw] [bw in KB/s]
-------

job     Set NR  cgrp1  cgrp2  cgrp3  cgrp4  total  
---     --- --  -----------------------------------
drw     1   1   1529   1530   1531   1532   6122   
drw     1   2   521    931    1185   1224   3861   
drw     1   4   150    382    988    1588   3108

Host=localhost.localdomain Kernel=2.6.37-rc2-Block-+
GROUPMODE=1 NRGRP=4
DIR=/mnt/iostestmnt/fio DEV=/dev/sdb2
Workload=drw iosched=cfq Filesz=512M bs=32k
group_isolation=1 slice_idle=8 group_idle=8 quantum=8
=========================================================================
AVERAGE[drw] [bw in KB/s]
-------

job     Set NR  cgrp1  cgrp2  cgrp3  cgrp4  total  
---     --- --  -----------------------------------
drw     1   1   529    1243   1887   2778   6437   
drw     1   2   296    717    1259   1817   4089   
drw     1   4   164    403    937    1662   3166

job     Set NR  cgrp1  cgrp2  cgrp3  cgrp4  total  
---     --- --  -----------------------------------
brr     1   1   759    759    760    758    3036   
brr     1   2   224    549    849    903    2525   
brr     1   4   146    307    558    1090   2101

Host=localhost.localdomain Kernel=2.6.37-rc2-Block-+
GROUPMODE=1 NRGRP=4
DIR=/mnt/iostestmnt/fio DEV=/dev/sdb2
Workload=brr iosched=cfq Filesz=512M bs=32k
group_isolation=1 slice_idle=8 group_idle=8 quantum=8
=========================================================================
AVERAGE[brr] [bw in KB/s]
-------

job     Set NR  cgrp1  cgrp2  cgrp3  cgrp4  total  
---     --- --  -----------------------------------
brr     1   1   375    734    1080   1426   3615   
brr     1   2   168    418    767    1233   2586   
brr     1   4   151    333    581    1068   2133

job     Set NR  cgrp1  cgrp2  cgrp3  cgrp4  total  
---     --- --  -----------------------------------
bsr     1   1   8000   8015   8027   8159   32201  
bsr     1   2   3200   6302   9612   10018  29132  
bsr     1   4   2153   4408   7857   13590  28008

Host=localhost.localdomain Kernel=2.6.37-rc2-Block-+
GROUPMODE=1 NRGRP=4
DIR=/mnt/iostestmnt/fio DEV=/dev/sdb2
Workload=bsr iosched=cfq Filesz=512M bs=32k
group_isolation=1 slice_idle=8 group_idle=8 quantum=8
=========================================================================
AVERAGE[bsr] [bw in KB/s]
-------

job     Set NR  cgrp1  cgrp2  cgrp3  cgrp4  total  
---     --- --  -----------------------------------
bsr     1   1   6988   11575  17883  23389  59835  
bsr     1   2   6808   11453  15979  19838  54078  
bsr     1   4   5352   10986  14529  18189  49056

job     Set NR  cgrp1  cgrp2  cgrp3  cgrp4  total  
---     --- --  -----------------------------------
drr     1   1   736    737    737    735    2945   
drr     1   2   214    550    854    912    2530   
drr     1   4   148    312    591    1069   2120

Host=localhost.localdomain Kernel=2.6.37-rc2-Block-+
GROUPMODE=1 NRGRP=4
DIR=/mnt/iostestmnt/fio DEV=/dev/sdb2
Workload=drr iosched=cfq Filesz=512M bs=32k
group_isolation=1 slice_idle=8 group_idle=8 quantum=8
=========================================================================
AVERAGE[drr] [bw in KB/s]
-------

job     Set NR  cgrp1  cgrp2  cgrp3  cgrp4  total  
---     --- --  -----------------------------------
drr     1   1   371    722    1059   1423   3575   
drr     1   2   173    412    805    1258   2648   
drr     1   4   155    305    555    1075   2090

job     Set NR  cgrp1  cgrp2  cgrp3  cgrp4  total  
---     --- --  -----------------------------------
drw     1   1   1491   1491   1491   1491   5964   
drw     1   2   556    869    1082   1201   3708   
drw     1   4   160    335    830    1690   3015

Host=localhost.localdomain Kernel=2.6.37-rc2-Block-+
GROUPMODE=1 NRGRP=4
DIR=/mnt/iostestmnt/fio DEV=/dev/sdb2
Workload=drw iosched=cfq Filesz=512M bs=32k
group_isolation=1 slice_idle=8 group_idle=8 quantum=8
=========================================================================
AVERAGE[drw] [bw in KB/s]
-------

job     Set NR  cgrp1  cgrp2  cgrp3  cgrp4  total  
---     --- --  -----------------------------------
drw     1   1   545    1343   2016   2743   6647   
drw     1   2   268    696    1361   1942   4267   
drw     1   4   140    357    775    1880   3152