blk-cgroup: Allow creation of hierarchical cgroups
Subsystems (3)
RFC 1/1 blk-cgroup: Allow creation of hierarchical cgroups
o Allow hierarchical cgroup creation for blkio controller
o Currently we disallow it as both the io controller policies (throttling as well as proportion bandwidth) do not support hierarhical accounting and control. But the flip side is that blkio controller can not be used with libvirt as libvirt creates a cgroup hierarchy deeper than 1 level.
<top-level-cgroup-dir>/<controller>/libvirt/qemu/<virtual-machine-groups>
o So this patch will allow creation of cgroup hierarhcy but at the backend everything will be treated as flat. So if somebody created a an hierarchy like as follows.
root / \ test1 test2 | test3
CFQ and throttling will practically treat all groups at same level.
pivot / | \ \ root test1 test2 test3
o Once we have actual support for hierarchical accounting and control then we can introduce another cgroup tunable file "blkio.use_hierarchy" which will be 0 by default but if user wants to enforce hierarhical control then it can be set to 1. This way there should not be any ABI problems down the line.
o The only not so pretty part is introduction of extra file "use_hierarchy" down the line. Kame-san had mentioned that hierarhical accounting is expensive in memory controller hence they keep it off by default. I suspect same will be the case for IO controller also as for each IO completion we shall have to account IO through hierarchy up to the root. if yes, then it probably is not a very bad idea to introduce this extra file so that it will be used only when somebody needs it and some people might enable hierarchy only in part of the hierarchy.
o This is how basically memory controller also uses "use_hierarhcy" and they also allowed creation of hierarchies when actual backend support was not available.
--- Documentation/cgroups/blkio-controller.txt | 27 +++++++++++++++++++++++++++ block/blk-cgroup.c | 4 ---- 2 files changed, 27 insertions(+), 4 deletions(-)
block/blk-cgroup.c +0 -4
@@ -1452,10 +1452,6 @@ blkiocg_create(struct cgroup_subsys *sub goto done; } -/* Currently we do not support hierarchy deeper than two level (0,1) */ -if (parent != cgroup->top_cgroup) -return ERR_PTR (-EPERM ); - blkcg =kzalloc (sizeof (*blkcg),GFP_KERNEL );if (!blkcg)return ERR_PTR (-ENOMEM );
Documentation/cgroups/blkio-controller.txt +27 -0
@@ -89,6 +89,33 @@ Throttling/Upper Limit policy Limits for writes can be put using blkio.write_bps_device file. +Hierarchical Cgroups +==================== +- Currently none of the IO control policy supports hierarhical groups. But + cgroup interface does allow creation of hierarhical cgroups and internally + IO policies treat them as flat hierarchy. + + So this patch will allow creation of cgroup hierarhcy but at the backend + everything will be treated as flat. So if somebody created a hierarchy like + as follows. + + root + / \ + test1 test2 + | + test3 + + CFQ and throttling will practically treat all groups at same level. + + pivot + / | \ \ + root test1 test2 test3 + + Down the line we can implement hierarchical accounting/control support + and also introduce a new cgroup file "use_hierarchy" which will control + whether cgroup hierarchy is viewed as flat or hierarchical by the policy. + This is how memory controller also has implemented the things. + Various user visible config options =================================== CONFIG_BLK_CGROUP
Quoted 1 line · Vivek Goyal
o Allow hierarchical cgroup creation for blkio controller
Vivek,
This patch looks fine, and we use similar code. I'd be even more interested in directing effort at Gui's patchset that fully implements hierarchy. If there are specific details that are blocking adoption of that code, let's address them.
Chad
Quoted 40 lines · Vivek Goyal
o Allow hierarchical cgroup creation for blkio controller
o Currently we disallow it as both the io controller policies (throttling as well as proportion bandwidth) do not support hierarhical accounting and control. But the flip side is that blkio controller can not be used with libvirt as libvirt creates a cgroup hierarchy deeper than 1 level.
<top-level-cgroup-dir>/<controller>/libvirt/qemu/<virtual-machine-groups>
o So this patch will allow creation of cgroup hierarhcy but at the backend everything will be treated as flat. So if somebody created a an hierarchy like as follows.
root / \ test1 test2 | test3CFQ and throttling will practically treat all groups at same level.
pivot / | \ \ root test1 test2 test3o Once we have actual support for hierarchical accounting and control then we can introduce another cgroup tunable file "blkio.use_hierarchy" which will be 0 by default but if user wants to enforce hierarhical control then it can be set to 1. This way there should not be any ABI problems down the line.
o The only not so pretty part is introduction of extra file "
use_hierarchy" down the line. Kame-san had mentioned that hierarhical accounting is expensive in memory controller hence they keep it off by default. I suspect same will be the case for IO controller also as for each IO completion we shall have to account IO through hierarchy up to the root. if yes, then it probably is not a very bad idea to introduce this extra file so that it will be used only when somebody needs it and some people might enable hierarchy only in part of the hierarchy.
Quoted 5 lines
o This is how basically memory controller also uses "
use_hierarhcy" and they also allowed creation of hierarchies when actual backend support was not available.Signed-off-by Vivek Goyal <[email protected]>
Signature · 2 lines
Three Cheers, Balbir
Quoted 40 lines · Vivek Goyal
o Allow hierarchical cgroup creation for blkio controller
o Currently we disallow it as both the io controller policies (throttling as well as proportion bandwidth) do not support hierarhical accounting and control. But the flip side is that blkio controller can not be used with libvirt as libvirt creates a cgroup hierarchy deeper than 1 level.
<top-level-cgroup-dir>/<controller>/libvirt/qemu/<virtual-machine-groups>
o So this patch will allow creation of cgroup hierarhcy but at the backend everything will be treated as flat. So if somebody created a an hierarchy like as follows.
root / \ test1 test2 | test3CFQ and throttling will practically treat all groups at same level.
pivot / | \ \ root test1 test2 test3o Once we have actual support for hierarchical accounting and control then we can introduce another cgroup tunable file "blkio.use_hierarchy" which will be 0 by default but if user wants to enforce hierarhical control then it can be set to 1. This way there should not be any ABI problems down the line.
o The only not so pretty part is introduction of extra file "
use_hierarchy" down the line. Kame-san had mentioned that hierarhical accounting is expensive in memory controller hence they keep it off by default. I suspect same will be the case for IO controller also as for each IO completion we shall have to account IO through hierarchy up to the root. if yes, then it probably is not a very bad idea to introduce this extra file so that it will be used only when somebody needs it and some people might enable hierarchy only in part of the hierarchy.
Quoted 5 lines
o This is how basically memory controller also uses "
use_hierarhcy" and they also allowed creation of hierarchies when actual backend support was not available.Signed-off-by Vivek Goyal <[email protected]>
Hi Vivek,
This patch looks good to me.
Quoted 59 lines
--- Documentation/cgroups/blkio-controller.txt | 27 +++++++++++++++++++++++++++ block/blk-cgroup.c | 4 ---- 2 files changed, 27 insertions(+), 4 deletions(-)Index:
linux-2.6/block/blk-cgroup.c
===================================================================block/blk-cgroup.c +0
-4@@ -1452,10 +1452,6 @@ blkiocg_create(struct cgroup_subsys *sub goto done; } -/* Currently we do not support hierarchy deeper than two level (0,1) */ -if (parent != cgroup->top_cgroup) -return ERR_PTR (-EPERM ); - blkcg =kzalloc (sizeof (*blkcg),GFP_KERNEL );if (!blkcg)return ERR_PTR (-ENOMEM );Index:
linux-2.6/Documentation/cgroups/blkio-controller.txt
===================================================================Documentation/cgroups/blkio-controller.txt +27
-0@@ -89,6 +89,33 @@ Throttling/Upper Limit policy Limits for writes can be put using blkio.write_bps_device file. +Hierarchical Cgroups +==================== +- Currently none of the IO control policy supports hierarhical groups. But + cgroup interface does allow creation of hierarhical cgroups and internally + IO policies treat them as flat hierarchy. + + So this patch will allow creation of cgroup hierarhcy but at the backend + everything will be treated as flat. So if somebody created a hierarchy like + as follows. + + root + / \ + test1 test2 + | + test3 + + CFQ and throttling will practically treat all groups at same level. + + pivot + / | \ \ + root test1 test2 test3 + + Down the line we can implement hierarchical accounting/control support + and also introduce a new cgroup file "use_hierarchy" which will control + whether cgroup hierarchy is viewed as flat or hierarchical by the policy. + This is how memory controller also has implemented the things. + Various user visible config options =================================== CONFIG_BLK_CGROUP
Quoted 9 lines · Chad Talbott
Quoted 1 line · Vivek Goyal
o Allow hierarchical cgroup creation for blkio controller
Vivek,
This patch looks fine, and we use similar code. I'd be even more interested in directing effort at Gui's patchset that fully implements hierarchy. If there are specific details that are blocking adoption of that code, let's address them.
Hi Chad,
My primary concern was re-organizing the code in such a way so that we treat queues and groups at the same level. I think Gui is now addressing those concerns.
I have detailed my concerns in this thread. (You are CCed though in the thread).
http://lkml.org/lkml/2010/8/30/30
Thanks
Vivek
Quoted 94 lines · Vivek Goyal
o Allow hierarchical cgroup creation for blkio controller
o Currently we disallow it as both the io controller policies (throttling as well as proportion bandwidth) do not support hierarhical accounting and control. But the flip side is that blkio controller can not be used with libvirt as libvirt creates a cgroup hierarchy deeper than 1 level.
<top-level-cgroup-dir>/<controller>/libvirt/qemu/<virtual-machine-groups>
o So this patch will allow creation of cgroup hierarhcy but at the backend everything will be treated as flat. So if somebody created a an hierarchy like as follows.
root / \ test1 test2 | test3CFQ and throttling will practically treat all groups at same level.
pivot / | \ \ root test1 test2 test3o Once we have actual support for hierarchical accounting and control then we can introduce another cgroup tunable file "blkio.use_hierarchy" which will be 0 by default but if user wants to enforce hierarhical control then it can be set to 1. This way there should not be any ABI problems down the line.
o The only not so pretty part is introduction of extra file "
use_hierarchy" down the line. Kame-san had mentioned that hierarhical accounting is expensive in memory controller hence they keep it off by default. I suspect same will be the case for IO controller also as for each IO completion we shall have to account IO through hierarchy up to the root. if yes, then it probably is not a very bad idea to introduce this extra file so that it will be used only when somebody needs it and some people might enable hierarchy only in part of the hierarchy.o This is how basically memory controller also uses "
use_hierarhcy" and they also allowed creation of hierarchies when actual backend support was not available.Signed-off-by Vivek Goyal <[email protected]>--- Documentation/cgroups/blkio-controller.txt | 27 +++++++++++++++++++++++++++ block/blk-cgroup.c | 4 ---- 2 files changed, 27 insertions(+), 4 deletions(-)Index:
linux-2.6/block/blk-cgroup.c
===================================================================block/blk-cgroup.c +0
-4@@ -1452,10 +1452,6 @@ blkiocg_create(struct cgroup_subsys *sub goto done; } -/* Currently we do not support hierarchy deeper than two level (0,1) */ -if (parent != cgroup->top_cgroup) -return ERR_PTR (-EPERM ); - blkcg =kzalloc (sizeof (*blkcg),GFP_KERNEL );if (!blkcg)return ERR_PTR (-ENOMEM );Index:
linux-2.6/Documentation/cgroups/blkio-controller.txt
===================================================================Documentation/cgroups/blkio-controller.txt +20
-0@@ -89,6 +89,33 @@ Throttling/Upper Limit policy Limits for writes can be put using blkio.write_bps_device file. +Hierarchical Cgroups +==================== +- Currently none of the IO control policy supports hierarhical groups. But + cgroup interface does allow creation of hierarhical cgroups and internally + IO policies treat them as flat hierarchy. + + So this patch will allow creation of cgroup hierarhcy but at the backend + everything will be treated as flat. So if somebody created a hierarchy like + as follows. + + root + / \ + test1 test2 + | + test3 + + CFQ and throttling will practically treat all groups at same level. + + pivot + / | \ \
Documentation/cgroups/blkio-controller.txt Throttling/Upper Limit policy · line 121
Quoted 10 lines
Documentation/cgroups/blkio-controller.txt +7
-0+ root test1 test2 test3 + + Down the line we can implement hierarchical accounting/control support + and also introduce a new cgroup file "use_hierarchy" which will control + whether cgroup hierarchy is viewed as flat or hierarchical by the policy. + This is how memory controller also has implemented the things. + Various user visible config options =================================== CONFIG_BLK_CGROUP
Quoted 7 lines
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Quoted 40 lines · Vivek Goyal
o Allow hierarchical cgroup creation for blkio controller
o Currently we disallow it as both the io controller policies (throttling as well as proportion bandwidth) do not support hierarhical accounting and control. But the flip side is that blkio controller can not be used with libvirt as libvirt creates a cgroup hierarchy deeper than 1 level.
<top-level-cgroup-dir>/<controller>/libvirt/qemu/<virtual-machine-groups>
o So this patch will allow creation of cgroup hierarhcy but at the backend everything will be treated as flat. So if somebody created a an hierarchy like as follows.
root / \ test1 test2 | test3CFQ and throttling will practically treat all groups at same level.
pivot / | \ \ root test1 test2 test3o Once we have actual support for hierarchical accounting and control then we can introduce another cgroup tunable file "blkio.use_hierarchy" which will be 0 by default but if user wants to enforce hierarhical control then it can be set to 1. This way there should not be any ABI problems down the line.
o The only not so pretty part is introduction of extra file "
use_hierarchy" down the line. Kame-san had mentioned that hierarhical accounting is expensive in memory controller hence they keep it off by default. I suspect same will be the case for IO controller also as for each IO completion we shall have to account IO through hierarchy up to the root. if yes, then it probably is not a very bad idea to introduce this extra file so that it will be used only when somebody needs it and some people might enable hierarchy only in part of the hierarchy.
Quoted 6 lines
o This is how basically memory controller also uses "
use_hierarhcy" and they also allowed creation of hierarchies when actual backend support was not available.Signed-off-by Vivek Goyal <[email protected]>---
Hi Jens,
Do you have any concerns about this patch? If not, can you please apply it.
Thanks
Vivek
Quoted 57 lines
Documentation/cgroups/blkio-controller.txt | 27 +++++++++++++++++++++++++++ block/blk-cgroup.c | 4 ---- 2 files changed, 27 insertions(+), 4 deletions(-)Index:
linux-2.6/block/blk-cgroup.c
===================================================================block/blk-cgroup.c +0
-4@@ -1452,10 +1452,6 @@ blkiocg_create(struct cgroup_subsys *sub goto done; } -/* Currently we do not support hierarchy deeper than two level (0,1) */ -if (parent != cgroup->top_cgroup) -return ERR_PTR (-EPERM ); - blkcg =kzalloc (sizeof (*blkcg),GFP_KERNEL );if (!blkcg)return ERR_PTR (-ENOMEM );Index:
linux-2.6/Documentation/cgroups/blkio-controller.txt
===================================================================Documentation/cgroups/blkio-controller.txt +27
-0@@ -89,6 +89,33 @@ Throttling/Upper Limit policy Limits for writes can be put using blkio.write_bps_device file. +Hierarchical Cgroups +==================== +- Currently none of the IO control policy supports hierarhical groups. But + cgroup interface does allow creation of hierarhical cgroups and internally + IO policies treat them as flat hierarchy. + + So this patch will allow creation of cgroup hierarhcy but at the backend + everything will be treated as flat. So if somebody created a hierarchy like + as follows. + + root + / \ + test1 test2 + | + test3 + + CFQ and throttling will practically treat all groups at same level. + + pivot + / | \ \ + root test1 test2 test3 + + Down the line we can implement hierarchical accounting/control support + and also introduce a new cgroup file "use_hierarchy" which will control + whether cgroup hierarchy is viewed as flat or hierarchical by the policy. + This is how memory controller also has implemented the things. + Various user visible config options =================================== CONFIG_BLK_CGROUP
Jens Axboe
AppliedQuoted 45 lines · Vivek Goyal
Quoted 44 lines · Vivek Goyal
o Allow hierarchical cgroup creation for blkio controller
o Currently we disallow it as both the io controller policies (throttling as well as proportion bandwidth) do not support hierarhical accounting and control. But the flip side is that blkio controller can not be used with libvirt as libvirt creates a cgroup hierarchy deeper than 1 level.
<top-level-cgroup-dir>/<controller>/libvirt/qemu/<virtual-machine-groups>
o So this patch will allow creation of cgroup hierarhcy but at the backend everything will be treated as flat. So if somebody created a an hierarchy like as follows.
root / \ test1 test2 | test3CFQ and throttling will practically treat all groups at same level.
pivot / | \ \ root test1 test2 test3o Once we have actual support for hierarchical accounting and control then we can introduce another cgroup tunable file "blkio.use_hierarchy" which will be 0 by default but if user wants to enforce hierarhical control then it can be set to 1. This way there should not be any ABI problems down the line.
o The only not so pretty part is introduction of extra file "
use_hierarchy" down the line. Kame-san had mentioned that hierarhical accounting is expensive in memory controller hence they keep it off by default. I suspect same will be the case for IO controller also as for each IO completion we shall have to account IO through hierarchy up to the root.. if yes, then it probably is not a very bad idea to introduce this extra file so that it will be used only when somebody needs it and some people might enable hierarchy only in part of the hierarchy.o This is how basically memory controller also uses "
use_hierarhcy" and they also allowed creation of hierarchies when actual backend support was not available.
Quoted 7 lines
Quoted 2 lines
Signed-off-by Vivek Goyal <[email protected]>---
Hi Jens,
Do you have any concerns about this patch? If not, can you please apply it.
Applied to for-2.6.38/rc2-holder, it'll be merged into for-2.6.38/core once -rc2 has been tagged (and I can pull in the conflicting bits).
Signature · 1 line
Jens Axboe
Quoted 40 lines · On Tue, 2 Nov 2010 18:20:30 -0400 Vivek Goyal <[email protected]> wrote:
o Allow hierarchical cgroup creation for blkio controller
o Currently we disallow it as both the io controller policies (throttling as well as proportion bandwidth) do not support hierarhical accounting and control. But the flip side is that blkio controller can not be used with libvirt as libvirt creates a cgroup hierarchy deeper than 1 level.
<top-level-cgroup-dir>/<controller>/libvirt/qemu/<virtual-machine-groups>
o So this patch will allow creation of cgroup hierarhcy but at the backend everything will be treated as flat. So if somebody created a an hierarchy like as follows.
root / \ test1 test2 | test3CFQ and throttling will practically treat all groups at same level.
pivot / | \ \ root test1 test2 test3o Once we have actual support for hierarchical accounting and control then we can introduce another cgroup tunable file "blkio.use_hierarchy" which will be 0 by default but if user wants to enforce hierarhical control then it can be set to 1. This way there should not be any ABI problems down the line.
o The only not so pretty part is introduction of extra file "
use_hierarchy" down the line. Kame-san had mentioned that hierarhical accounting is expensive in memory controller hence they keep it off by default. I suspect same will be the case for IO controller also as for each IO completion we shall have to account IO through hierarchy up to the root. if yes, then it probably is not a very bad idea to introduce this extra file so that it will be used only when somebody needs it and some people might enable hierarchy only in part of the hierarchy.
Quoted 5 lines
o This is how basically memory controller also uses "
use_hierarhcy" and they also allowed creation of hierarchies when actual backend support was not available.Signed-off-by Vivek Goyal <[email protected]>
Thank you!
Quoted 59 lines
--- Documentation/cgroups/blkio-controller.txt | 27 +++++++++++++++++++++++++++ block/blk-cgroup.c | 4 ---- 2 files changed, 27 insertions(+), 4 deletions(-)Index:
linux-2.6/block/blk-cgroup.c
===================================================================block/blk-cgroup.c +0
-4@@ -1452,10 +1452,6 @@ blkiocg_create(struct cgroup_subsys *sub goto done; } -/* Currently we do not support hierarchy deeper than two level (0,1) */ -if (parent != cgroup->top_cgroup) -return ERR_PTR (-EPERM ); - blkcg =kzalloc (sizeof (*blkcg),GFP_KERNEL );if (!blkcg)return ERR_PTR (-ENOMEM );Index:
linux-2.6/Documentation/cgroups/blkio-controller.txt
===================================================================Documentation/cgroups/blkio-controller.txt +27
-0@@ -89,6 +89,33 @@ Throttling/Upper Limit policy Limits for writes can be put using blkio.write_bps_device file. +Hierarchical Cgroups +==================== +- Currently none of the IO control policy supports hierarhical groups. But + cgroup interface does allow creation of hierarhical cgroups and internally + IO policies treat them as flat hierarchy. + + So this patch will allow creation of cgroup hierarhcy but at the backend + everything will be treated as flat. So if somebody created a hierarchy like + as follows. + + root + / \ + test1 test2 + | + test3 + + CFQ and throttling will practically treat all groups at same level. + + pivot + / | \ \ + root test1 test2 test3 + + Down the line we can implement hierarchical accounting/control support + and also introduce a new cgroup file "use_hierarchy" which will control + whether cgroup hierarchy is viewed as flat or hierarchical by the policy. + This is how memory controller also has implemented the things. + Various user visible config options =================================== CONFIG_BLK_CGROUP