cgroup: Show # of subsystem CSSes in cgroup.stat
Subsystems (2)
PATCH v6 1/1 cgroup: Show # of subsystem CSSes in cgroup.stat
Cgroup subsystem state (CSS) is an abstraction in the cgroup layer to help manage different structures in various cgroup subsystems by being an embedded element inside a larger structure like cpuset or mem_cgroup.
The /proc/cgroups file shows the number of cgroups for each of the subsystems. With cgroup v1, the number of CSSes is the same as the number of cgroups. That is not the case anymore with cgroup v2. The /proc/cgroups file cannot show the actual number of CSSes for the subsystems that are bound to cgroup v2.
So if a v2 cgroup subsystem is leaking cgroups (usually memory cgroup), we can't tell by looking at /proc/cgroups which cgroup subsystems may be responsible.
As cgroup v2 had deprecated the use of /proc/cgroups, the hierarchical cgroup.stat file is now being extended to show the number of live and dying CSSes associated with all the non-inhibited cgroup subsystems that have been bound to cgroup v2 as long as they are not both zero. The number includes CSSes in the current cgroup as well as in all the descendants underneath it. This will help us pinpoint which subsystems are responsible for the increasing number of dying (nr_dying_descendants) cgroups.
The CSSes dying counts are stored in the cgroup structure itself instead of inside the CSS as suggested by Johannes. This will allow us to accurately track dying counts of cgroup subsystems that have recently been disabled in a cgroup. It is now possible that a zero subsystem number is coupled with a non-zero dying subsystem number.
The cgroup-v2.rst file is updated to discuss this new behavior.
With this patch applied, a sample output from root cgroup.stat file was shown below.
nr_descendants 56 nr_subsys_cpuset 1 nr_subsys_cpu 43 nr_subsys_io 43 nr_subsys_memory 56 nr_subsys_perf_event 57 nr_subsys_hugetlb 1 nr_subsys_pids 56 nr_subsys_rdma 1 nr_subsys_misc 1 nr_dying_descendants 30 nr_dying_subsys_cpuset 0 nr_dying_subsys_cpu 0 nr_dying_subsys_io 0 nr_dying_subsys_memory 30 nr_dying_subsys_perf_event 0 nr_dying_subsys_hugetlb 0 nr_dying_subsys_pids 0 nr_dying_subsys_rdma 0 nr_dying_subsys_misc 0
Another sample output from system.slice/cgroup.stat was:
nr_descendants 32 nr_subsys_cpu 30 nr_subsys_io 30 nr_subsys_memory 32 nr_subsys_perf_event 33 nr_subsys_pids 32 nr_dying_descendants 32 nr_dying_subsys_cpu 0 nr_dying_subsys_io 0 nr_dying_subsys_memory 32 nr_dying_subsys_perf_event 0 nr_dying_subsys_pids 0
--- Documentation/admin-guide/cgroup-v2.rst | 16 ++++++- include/linux/cgroup-defs.h | 14 ++++++ kernel/cgroup/cgroup.c | 58 ++++++++++++++++++++++++- 3 files changed, 84 insertions(+), 4 deletions(-)
Documentation/admin-guide/cgroup-v2.rst +14 -2
@@ -981,6 +981,18 @@ All cgroup core files are prefixed with "cgroup." A dying cgroup can consume system resources not exceeding limits, which were active at the moment of cgroup deletion. + nr_subsys_<cgroup_subsys> + Total number of live cgroup subsystems (e.g memory + cgroup) at and beneath the current cgroup. An entry + will only be shown if the cgroup subsystem is enabled + in the current cgroup or dying subsystems are present. + + nr_dying_subsys_<cgroup_subsys> + Total number of dying cgroup subsystems (e.g. memory + cgroup) at and beneath the current cgroup. An entry + will only be shown if the cgroup subsystem is enabled + in the current cgroup or dying subsystems are present. + cgroup.freeze A read-write single value file which exists on non-root cgroups. Allowed values are "0" and "1". The default is "0".
@@ -2930,8 +2942,8 @@ Deprecated v1 Core Features - "cgroup.clone_children" is removed. -- /proc/cgroups is meaningless for v2. Use "cgroup.controllers" file - at the root instead. +- /proc/cgroups is meaningless for v2. Use "cgroup.controllers" or + "cgroup.stat" files at the root instead. Issues with v1 and Rationales for v2
include/linux/cgroup-defs.h +14 -0
@@ -210,6 +210,14 @@ struct cgroup_subsys_state {* fields of the containing structure. */ struct cgroup_subsys_state *parent; + +/* +* Keep track of total numbers of visible descendant CSSes. +* The total number of dying CSSes is tracked in +* css->cgroup->nr_dying_subsys[ssid]. +* Protected by cgroup_mutex. +*/ +int nr_descendants; };/*
@@ -470,6 +478,12 @@ struct cgroup {/* Private pointers for each registered subsystem */ struct cgroup_subsys_state __rcu *subsys[CGROUP_SUBSYS_COUNT ]; +/* +* Keep track of total number of dying CSSes at and below this cgroup. +* Protected by cgroup_mutex. +*/ +int nr_dying_subsys[CGROUP_SUBSYS_COUNT ]; +struct cgroup_root *root;/*
kernel/cgroup/cgroup.c +56 -2
94 lines not on this page · raw message
Signature · 1 line
2.39.3
Quoted 70 lines · Waiman Long
Cgroup subsystem state (CSS) is an abstraction in the cgroup layer to help manage different structures in various cgroup subsystems by being an embedded element inside a larger structure like cpuset or
mem_cgroup.The
/proc/cgroupsfile shows the number of cgroups for each of the subsystems. With cgroup v1, the number of CSSes is the same as the number of cgroups. That is not the case anymore with cgroup v2. The/proc/cgroupsfile cannot show the actual number of CSSes for the subsystems that are bound to cgroup v2.So if a v2 cgroup subsystem is leaking cgroups (usually memory cgroup), we can't tell by looking at
/proc/cgroupswhich cgroup subsystems may be responsible.As cgroup v2 had deprecated the use of
/proc/cgroups, the hierarchical cgroup.stat file is now being extended to show the number of live and dying CSSes associated with all the non-inhibited cgroup subsystems that have been bound to cgroup v2 as long as they are not both zero. The number includes CSSes in the current cgroup as well as in all the descendants underneath it. This will help us pinpoint which subsystems are responsible for the increasing number of dying (nr_dying_descendants) cgroups.The CSSes dying counts are stored in the cgroup structure itself instead of inside the CSS as suggested by Johannes. This will allow us to accurately track dying counts of cgroup subsystems that have recently been disabled in a cgroup. It is now possible that a zero subsystem number is coupled with a non-zero dying subsystem number.
The
cgroup-v2.rstfile is updated to discuss this new behavior.With this patch applied, a sample output from root cgroup.stat file was shown below.
nr_descendants 56 nr_subsys_cpuset 1 nr_subsys_cpu 43 nr_subsys_io 43 nr_subsys_memory 56 nr_subsys_perf_event 57 nr_subsys_hugetlb 1 nr_subsys_pids 56 nr_subsys_rdma 1 nr_subsys_misc 1 nr_dying_descendants 30 nr_dying_subsys_cpuset 0 nr_dying_subsys_cpu 0 nr_dying_subsys_io 0 nr_dying_subsys_memory 30 nr_dying_subsys_perf_event 0 nr_dying_subsys_hugetlb 0 nr_dying_subsys_pids 0 nr_dying_subsys_rdma 0 nr_dying_subsys_misc 0Another sample output from system.slice/cgroup.stat was:
nr_descendants 32 nr_subsys_cpu 30 nr_subsys_io 30 nr_subsys_memory 32 nr_subsys_perf_event 33 nr_subsys_pids 32 nr_dying_descendants 32 nr_dying_subsys_cpu 0 nr_dying_subsys_io 0 nr_dying_subsys_memory 32 nr_dying_subsys_perf_event 0 nr_dying_subsys_pids 0
[...]
Quoted 21 lines
kernel/cgroup/cgroup.c +10
-0@@ -3669,12 +3669,43 @@ static int cgroup_events_show(struct seq_file *seq, void *v)static int cgroup_stat_show (struct seq_file *seq,void *v) {struct cgroup *cgroup =seq_css (seq)->cgroup; +struct cgroup_subsys_state *css; +int dying_cnt[CGROUP_SUBSYS_COUNT ]; +int ssid;seq_printf (seq,"nr_descendants %d\n" , cgroup->nr_descendants); + +/* +* Show the number of live and dying csses associated with each of +* non-inhibited cgroup subsystems that is either enabled in current +* cgroup or has non-zero dying count. +* +* Without proper lock protection, racing is possible. So the
kernel/cgroup/cgroup.c static int cgroup_events_show(struct seq_file *seq, void *v) · line 3694
Quoted 10 lines
kernel/cgroup/cgroup.c +10
-0+* numbers may not be consistent when that happens. +*/ +rcu_read_lock (); +for (ssid =0 ; ssid <CGROUP_SUBSYS_COUNT ; ssid++) { + dying_cnt[ssid] =-1 ; +if (BIT (ssid) & cgrp_dfl_inhibit_ss_mask) +continue ; + css =rcu_dereference_raw (cgroup->subsys[ssid]); +if (!css && !cgroup->nr_dying_subsys[ssid]) +continue ;
Sorry, If I have misread the discussion from the other thread about displaying nr_descendants and nr_dying_subsys_<subsys>. I believe the idea was to print them for enabled and disabled cgroup controllers, so the output stays consistent and does not vary depending on the enabled controllers or previously enabled controller with nr_dying_subsys > 0.
For example, on a rebooted vm:
# cd /sys/fs/cgroup/ # cat cgroup.subtree_control cpu memory pids
# mkdir foo # cat foo/cgroup.stat nr_descendants 0 nr_subsys_cpu 1 nr_subsys_memory 1 nr_subsys_perf_event 1 nr_subsys_pids 1 nr_dying_descendants 0 nr_dying_subsys_cpu 0 nr_dying_subsys_memory 0 nr_dying_subsys_perf_event 0 nr_dying_subsys_pids 0
# echo '+cpuset' > cgroup.subtree_control
# cat foo/cgroup.stat nr_descendants 0 nr_subsys_cpuset 1 nr_subsys_cpu 1 nr_subsys_memory 1 nr_subsys_perf_event 1 nr_subsys_pids 1 nr_dying_descendants 0 nr_dying_subsys_cpuset 0 nr_dying_subsys_cpu 0 nr_dying_subsys_memory 0 nr_dying_subsys_perf_event 0 nr_dying_subsys_pids 0
Quoted 18 lines
kernel/cgroup/cgroup.c +12
-1+ + dying_cnt[ssid] = cgroup->nr_dying_subsys[ssid]; +seq_printf (seq,"nr_subsys_%s %d\n" , cgroup_subsys[ssid]->name, + css ? (css->nr_descendants +1 ) :0 ); + } +seq_printf (seq,"nr_dying_descendants %d\n" , cgroup->nr_dying_descendants); - +for (ssid =0 ; ssid <CGROUP_SUBSYS_COUNT ; ssid++) { +if (dying_cnt[ssid] >=0 ) +seq_printf (seq,"nr_dying_subsys_%s %d\n" , + cgroup_subsys[ssid]->name, dying_cnt[ssid]); + } +rcu_read_unlock ();return 0 ; }
Signature · 2 lines
Thanks, Kamalesh
Quoted 131 lines · Kamalesh Babulal
Quoted 70 lines · Waiman Long
Cgroup subsystem state (CSS) is an abstraction in the cgroup layer to help manage different structures in various cgroup subsystems by being an embedded element inside a larger structure like cpuset or
mem_cgroup.The
/proc/cgroupsfile shows the number of cgroups for each of the subsystems. With cgroup v1, the number of CSSes is the same as the number of cgroups. That is not the case anymore with cgroup v2. The/proc/cgroupsfile cannot show the actual number of CSSes for the subsystems that are bound to cgroup v2.So if a v2 cgroup subsystem is leaking cgroups (usually memory cgroup), we can't tell by looking at
/proc/cgroupswhich cgroup subsystems may be responsible.As cgroup v2 had deprecated the use of
/proc/cgroups, the hierarchical cgroup.stat file is now being extended to show the number of live and dying CSSes associated with all the non-inhibited cgroup subsystems that have been bound to cgroup v2 as long as they are not both zero. The number includes CSSes in the current cgroup as well as in all the descendants underneath it. This will help us pinpoint which subsystems are responsible for the increasing number of dying (nr_dying_descendants) cgroups.The CSSes dying counts are stored in the cgroup structure itself instead of inside the CSS as suggested by Johannes. This will allow us to accurately track dying counts of cgroup subsystems that have recently been disabled in a cgroup. It is now possible that a zero subsystem number is coupled with a non-zero dying subsystem number.
The
cgroup-v2.rstfile is updated to discuss this new behavior.With this patch applied, a sample output from root cgroup.stat file was shown below.
nr_descendants 56 nr_subsys_cpuset 1 nr_subsys_cpu 43 nr_subsys_io 43 nr_subsys_memory 56 nr_subsys_perf_event 57 nr_subsys_hugetlb 1 nr_subsys_pids 56 nr_subsys_rdma 1 nr_subsys_misc 1 nr_dying_descendants 30 nr_dying_subsys_cpuset 0 nr_dying_subsys_cpu 0 nr_dying_subsys_io 0 nr_dying_subsys_memory 30 nr_dying_subsys_perf_event 0 nr_dying_subsys_hugetlb 0 nr_dying_subsys_pids 0 nr_dying_subsys_rdma 0 nr_dying_subsys_misc 0Another sample output from system.slice/cgroup.stat was:
nr_descendants 32 nr_subsys_cpu 30 nr_subsys_io 30 nr_subsys_memory 32 nr_subsys_perf_event 33 nr_subsys_pids 32 nr_dying_descendants 32 nr_dying_subsys_cpu 0 nr_dying_subsys_io 0 nr_dying_subsys_memory 32 nr_dying_subsys_perf_event 0 nr_dying_subsys_pids 0[...]
Quoted 21 lines
kernel/cgroup/cgroup.c +10
-0@@ -3669,12 +3669,43 @@ static int cgroup_events_show(struct seq_file *seq, void *v)static int cgroup_stat_show (struct seq_file *seq,void *v) {struct cgroup *cgroup =seq_css (seq)->cgroup; +struct cgroup_subsys_state *css; +int dying_cnt[CGROUP_SUBSYS_COUNT ]; +int ssid;seq_printf (seq,"nr_descendants %d\n" , cgroup->nr_descendants); + +/* +* Show the number of live and dying csses associated with each of +* non-inhibited cgroup subsystems that is either enabled in current +* cgroup or has non-zero dying count. +* +* Without proper lock protection, racing is possible. So the kernel/cgroup/cgroup.c static int cgroup_events_show(struct seq_file *seq, void *v) · line 3694
Quoted 10 lines
kernel/cgroup/cgroup.c +10
-0+* numbers may not be consistent when that happens. +*/ +rcu_read_lock (); +for (ssid =0 ; ssid <CGROUP_SUBSYS_COUNT ; ssid++) { + dying_cnt[ssid] =-1 ; +if (BIT (ssid) & cgrp_dfl_inhibit_ss_mask) +continue ; + css =rcu_dereference_raw (cgroup->subsys[ssid]); +if (!css && !cgroup->nr_dying_subsys[ssid]) +continue ;Sorry, If I have misread the discussion from the other thread about displaying
nr_descendantsand nr_dying_subsys_<subsys>. I believe the idea was to print them for enabled and disabled cgroup controllers, so the output stays consistent and does not vary depending on the enabled controllers or previously enabled controller withnr_dying_subsys> 0.For example, on a rebooted vm:
# cd /sys/fs/cgroup/ # cat cgroup.subtree_control cpu memory pids# mkdir foo # cat foo/cgroup.stat nr_descendants 0 nr_subsys_cpu 1 nr_subsys_memory 1 nr_subsys_perf_event 1 nr_subsys_pids 1 nr_dying_descendants 0 nr_dying_subsys_cpu 0 nr_dying_subsys_memory 0 nr_dying_subsys_perf_event 0 nr_dying_subsys_pids 0# echo '+cpuset' > cgroup.subtree_control
Quoted 13 lines
# cat foo/cgroup.stat nr_descendants 0 nr_subsys_cpuset 1 nr_subsys_cpu 1 nr_subsys_memory 1 nr_subsys_perf_event 1 nr_subsys_pids 1 nr_dying_descendants 0 nr_dying_subsys_cpuset 0 nr_dying_subsys_cpu 0 nr_dying_subsys_memory 0 nr_dying_subsys_perf_event 0 nr_dying_subsys_pids 0
I am fine with fine with that. I will update the patch as suggested.
Thanks,
Longman
Quoted 18 lines
Quoted 18 lines
kernel/cgroup/cgroup.c +12
-1+ + dying_cnt[ssid] = cgroup->nr_dying_subsys[ssid]; +seq_printf (seq,"nr_subsys_%s %d\n" , cgroup_subsys[ssid]->name, + css ? (css->nr_descendants +1 ) :0 ); + } +seq_printf (seq,"nr_dying_descendants %d\n" , cgroup->nr_dying_descendants); - +for (ssid =0 ; ssid <CGROUP_SUBSYS_COUNT ; ssid++) { +if (dying_cnt[ssid] >=0 ) +seq_printf (seq,"nr_dying_subsys_%s %d\n" , + cgroup_subsys[ssid]->name, dying_cnt[ssid]); + } +rcu_read_unlock ();return 0 ; }