cgroup: Introducing bypass mode
Subsystems (4)
v5 cover cgroup: Introducing bypass mode
v5:
- Fix a bug in css_create() reported by Dan Carpenter.
v4:
- Rebased to the latest for-4.21 branch of cgroup tree.
- Make each controller explicitly opt in to become bypassable by setting the bypassable cgroup_subsys flag. Currently, only cpu controller is made bypassable.
- Break out the cgroup_v2.rst documentation update as separate patch.
v3:
- Remove invalid cgroup subdirectory creation patch.
- Add use cases for the bypass mode and removing statements about control files ownership in cgroup-v2.txt.
- Restrict bypass mode to non-domain (threaded) controllers only.
v3 patch - https://lkml.org/lkml/2017/8/9/604
v4 patch - https://lkml.org/lkml/2018/11/20/1361
This patchset introduces a new bypass mode to the cgroup v2 core to give more freedom and flexibility to controllers which choose to become bypassable the freedom to shape their own unique views of the virtual cgroup hierarchies that can best suit thier own use cases.
Because of the inherent performance overhead in enabling cpu controller, it is made bypassable so that the controller only needs to be enabled at those cgroups that really need it instead of in every cgroups at a given layer if at least one of them needs it.
The cpu controller performance problem is one of the major issues in migrating from cgroup v1 to v2.
For example,
R - A(+) - B(#) - C(+)
\ D(#)
where "+" means the controller is enabled and "#" means the controller is bypassed. For this controller's perspective, the cgroups are equivalent to:
R - A|B|D - C
Underneath the root R, cgoups A, B and D are controlled by one set of knobs and cgroup C is controlled by another set of knobs as a child of cgroups A|B|C.
This patchset is layered on top of the "for-4.21" branch of Tejun's cgroup git tree.
Patch 1 introduces a new bypass mode that allows a bypassable controller to be disabled in a cgroup, but to be re-enabled again in its children. This is enabled by writing the controller name prefixed with '#' to the "cgroup.subtree_control" file. Then all its children will have this controller in bypass mode.
Patch 2 extends the bypass mode mechanism to allow those child cgroups that are put into the bypass mode for a particular bypassable controller by their parent to be re-enabled again by writing the controller name with the '+' prefix to the "cgroup.controllers" file.
Patch 3 extends the debug controller to expose additional controller masks introduced by this patchset.
Patch 4 makes the cpu controller bypassable.
Patch 5 documents the new bypass mode in cgroup-v2.rst file.
Waiman Long (5):
cgroup: subtree_control bypass mode for bypassable controllers
cgroup: Allow reenabling of controller in bypass mode
cgroup: Make debug controller report new controller masks sched/core: Make cpu cgroup controller bypassable
cgroup: Document bypass mode
Documentation/admin-guide/cgroup-v2.rst | 66 +++++--- include/linux/cgroup-defs.h | 26 +++- kernel/cgroup/cgroup.c | 258 +++++++++++++++++++++++++------- kernel/cgroup/debug.c | 2 + kernel/sched/core.c | 1 + 5 files changed, 279 insertions(+), 74 deletions(-)
Signature · 1 line
1.8.3.1
PATCH v5 1/5 cgroup: subtree_control bypass mode for bypassable controllers
A controller in the default hierarchy is bypassable if the bypassable flag is set in its cgroup_subsys structure. The special prefix '#' attached to a bypassable controller name can now be written into the cgroup.subtree_control file to set that controller in bypass mode in all the child cgroups. The controller will show up in the children's cgroup.controllers file, but the corresponding control knobs will be absent. However, that controller can be enabled or bypassed in its children by writing to their respective subtree_control files.
In term of resource control, a bypassed cgroup was lumped into its nearest non-bypassed ancestor. So all the tasks in that ancestor as well as those in its bypassed direct descendants are controlled by the same set of control knobs.
This mode is useful to those bypassable controllers where there are costs to each additional layer of hierarchy. This mode will also allow more freedom in how each controller can shape its effective hierarchy independent of the others.
--- include/linux/cgroup-defs.h | 19 ++++-- kernel/cgroup/cgroup.c | 151 ++++++++++++++++++++++++++++++-------------- 2 files changed, 116 insertions(+), 54 deletions(-)
include/linux/cgroup-defs.h +14 -5
@@ -375,16 +375,18 @@ struct cgroup {struct cgroup_file events_file;/* handle for "cgroup.events" */ /* -* The bitmask of subsystems enabled on the child cgroups. -* ->subtree_control is the one configured through -* "cgroup.subtree_control" while ->child_ss_mask is the effective -* one which may have more subsystems enabled. Controller knobs -* are made available iff it's enabled in ->subtree_control. +* The bitmask of subsystems enabled or bypassed on the child cgroups. +* ->subtree_control and ->subtree_bypass are the one configured +* through "cgroup.subtree_control" while ->subtree_ss_mask is the +* effective one which may have more subsystems enabled. Controller +* knobs are made available iff it's enabled in ->subtree_ss_mask. */ u16 subtree_control;u16 subtree_ss_mask; +u16 subtree_bypass;u16 old_subtree_control;u16 old_subtree_ss_mask; +u16 old_subtree_bypass;/* Private pointers for each registered subsystem */ struct cgroup_subsys_state __rcu *subsys[CGROUP_SUBSYS_COUNT ];
@@ -647,6 +649,13 @@ struct cgroup_subsys { bool broken_hierarchy:1; bool warned_broken_hierarchy:1; +/* +* If %true, the controller, on the default hierarchy, can be +* bypassable. IOW, it can have a virtual hierarchy that is +* different from the default hierarchy. +*/ + bool bypassable:1; +/* the following two fields are initialized automtically during boot */ int id;const char *name;
kernel/cgroup/cgroup.c +102 -49
344 lines not on this page · raw message
Signature · 1 line
1.8.3.1
PATCH v5 2/5 cgroup: Allow reenabling of controller in bypass mode
In reply to Waiman LongBypassable controllers set to bypass mode in the parent's "cgroup.subtree_control" can now be optionally enabled by writing the controller name with the '+' prefix to "cgroup.controllers". Using the '#' prefix will reset it back to the bypass state.
This capability allows a cgroup parent to individually enable bypassable controllers in a subset of its children instead of either all or none of them. This increases the flexibility each controller has in shaping the effective cgroup hierarchy to best suit its need.
--- include/linux/cgroup-defs.h | 7 +++ kernel/cgroup/cgroup.c | 109 ++++++++++++++++++++++++++++++++++++++++++-- 2 files changed, 113 insertions(+), 3 deletions(-)
include/linux/cgroup-defs.h +7 -0
@@ -388,6 +388,13 @@ struct cgroup {u16 old_subtree_ss_mask;u16 old_subtree_bypass; +/* +* The bitmask of subsystems that are set in its parent's +* ->subtree_bypass and explicitly enabled in this cgroup. +*/ +u16 enable_ss_mask; +u16 old_enable_ss_mask; +/* Private pointers for each registered subsystem */ struct cgroup_subsys_state __rcu *subsys[CGROUP_SUBSYS_COUNT ];
kernel/cgroup/cgroup.c +106 -3
151 lines not on this page · raw message
Signature · 1 line
1.8.3.1
PATCH v5 3/5 cgroup: Make debug controller report new controller masks
In reply to Waiman LongThe newly added cgroup controller masks (subtree_bypass and enable_ss_mask) are now being reported in the debug.masks controller file.
--- kernel/cgroup/debug.c | 2 ++ 1 file changed, 2 insertions(+)
kernel/cgroup/debug.c +2 -0
@@ -263,6 +263,8 @@ static int cgroup_masks_read(struct seq_file *seq, void *v)cgroup_masks_read_one (seq,"subtree_control" , cgrp->subtree_control);cgroup_masks_read_one (seq,"subtree_ss_mask" , cgrp->subtree_ss_mask); +cgroup_masks_read_one (seq,"subtree_bypass" , cgrp->subtree_bypass); +cgroup_masks_read_one (seq,"enable_ss_mask" , cgrp->enable_ss_mask);cgroup_kn_unlock (of->kn);return 0 ;
Signature · 1 line
1.8.3.1
PATCH v5 4/5 sched/core: Make cpu cgroup controller bypassable
In reply to Waiman LongMake the cpu cgroup controller bypassable in the default hierarchy so that cpu controller could be activated only in those cgroups that really need it instead of all the cgroups down to the lowest level that need it.
--- kernel/sched/core.c | 1 + 1 file changed, 1 insertion(+)
kernel/sched/core.c +1 -0
@@ -7000,6 +7000,7 @@ struct cgroup_subsys cpu_cgrp_subsys = { .dfl_cftypes = cpu_files, .early_init = true, .threaded = true, + .bypassable = true, };#endif /* CONFIG_CGROUP_SCHED */
Signature · 1 line
1.8.3.1
PATCH v5 5/5 cgroup: Document bypass mode
In reply to Waiman LongThe cgroup-v2.rst file is updated to document the new bypass mode.
--- Documentation/admin-guide/cgroup-v2.rst | 66 ++++++++++++++++++++++++--------- 1 file changed, 48 insertions(+), 18 deletions(-)
Documentation/admin-guide/cgroup-v2.rst +48 -18
@@ -372,9 +372,16 @@ disabled by writing to the "cgroup.subtree_control" file:: # echo "+cpu +memory -io" > cgroup.subtree_control -Only controllers which are listed in "cgroup.controllers" can be -enabled. When multiple operations are specified as above, either they -all succeed or fail. If multiple operations on the same controller +The prefixes '+', '-' and '#' are used to enable, disable or put +a controller in the bypass mode respectively. In the bypass mode, +a controller is disabled in a cgroup, but it can be enabled again in +its child cgroups as it will still be listed in "cgroup.controllers". +Bypass mode can only be used on bypassable controllers. Currently, +only the cpu controller is bypassable. + +Only controllers which are listed in "cgroup.controllers" can be enabled +or bypassed. When multiple operations are specified as above, either +they all succeed or fail. If multiple operations on the same controller are specified, the last one is effective. Enabling a controller in a cgroup indicates that the distribution of
@@ -399,6 +406,20 @@ prefixed controller interface files from C and D. This means that the controller interface files - anything which doesn't start with "cgroup." are owned by the parent rather than the cgroup itself. +Once a bypassable controller is put into bypass mode in +"cgroup.subtree_control", that controller can optionally be enabled +again in child cgroups by writing the controller name with the '+ prefix +into "cgroup.controllers". Writing the controller name with the '#' +prefix into "cgroup.controllers" resets the state back to bypass mode. +The state of a bypassable controller cannot be changed anymore if it +is enabled or bypassed in its "cgroup.subtree_control". + +The use of bypass mode thus allows a cgroup parent to have the ability +to selectively enable a bypassable controller in a subset of its child +cgroups instead of in either all or none of them. In other words, a +bypassable can be enabled only on the cgroups that actually need it, +if desired. + Top-down Constraint ~~~~~~~~~~~~~~~~~~~
@@ -406,10 +427,11 @@ Top-down Constraint Resources are distributed top-down and a cgroup can further distribute a resource only if the resource has been distributed to it from the parent. This means that all non-root "cgroup.subtree_control" files -can only contain controllers which are enabled in the parent's -"cgroup.subtree_control" file. A controller can be enabled only if -the parent has the controller enabled and a controller can't be -disabled if one or more children have it enabled. +can only contain controllers which are enabled or bypassed in the +parent's "cgroup.subtree_control" file. A controller can be enabled +or bypassed only if the parent has the controller enabled or bypassed +and the state of a controller can't be changed if one or more children +have it enabled or bypassed. No Internal Process Constraint
@@ -834,11 +856,18 @@ All cgroup core files are prefixed with "cgroup." should be granted along with the containing directory. cgroup.controllers - A read-only space separated values file which exists on all + A read-write space separated values file which exists on all cgroups. It shows space separated list of all controllers available to - the cgroup. The controllers are not ordered. + the cgroup. Controller names with '#' prefix are in bypass mode. + The controllers are not ordered. + + When a controller is set into bypass mode in its parent's + "cgroup.subtree_control", its name prefixed with '+' or '#' + can be written to enable it or reset it back to bypass mode + respectively. Controllers not in bypass mode are not allowed + to be written. cgroup.subtree_control A read-write space separated values file which exists on all
@@ -848,12 +877,12 @@ All cgroup core files are prefixed with "cgroup." which are enabled to control resource distribution from the cgroup to its children. - Space separated list of controllers prefixed with '+' or '-' - can be written to enable or disable controllers. A controller - name prefixed with '+' enables the controller and '-' - disables. If a controller appears more than once on the list, - the last one is effective. When multiple enable and disable - operations are specified, either all succeed or all fail. + Space separated list of controllers prefixed with '+', '-' or + '#' can be written to enable, disable or bypass controllers + respectively. If a controller appears more than once on + the list, the last one is effective. When multiple enable, + disable or bypass operations are specified, either all succeed + or all fail. cgroup.events A read-only flat-keyed file which exists on non-root cgroups.
@@ -904,17 +933,18 @@ Controllers CPU --- -The "cpu" controllers regulates distribution of CPU cycles. This +The "cpu" controller regulates distribution of CPU cycles. This controller implements weight and absolute bandwidth limit models for normal scheduling policy and absolute bandwidth allocation model for -realtime scheduling policy. +realtime scheduling policy. The cpu controller is bypassable. WARNING: cgroup2 doesn't yet support control of realtime processes and the cpu controller can only be enabled when all RT processes are in the root cgroup. Be aware that system management software may already have placed RT processes into nonroot cgroups during the system boot process, and these processes may need to be moved to the root cgroup -before the cpu controller can be enabled. +before the cpu controller can be enabled unless bypass mode is used +in those non-root cgroups. CPU Interface Files
Signature · 1 line
1.8.3.1