cgroup: Track time in cgroup v2 freezer
Subsystems (2)
RFC PATCH v2 1/1 cgroup: Track time in cgroup v2 freezer
The cgroup v2 freezer controller allows user processes to be dynamically added to and removed from an interruptible frozen state from userspace. This feature is helpful for application management, as a background app can be frozen to prevent its threads from being scheduled or otherwise contending with foreground tasks for resources. However, the application is usually unaware that it was frozen, which can cause issues by disrupting any internal monitoring that it is performing.
As an example, an application may implement a watchdog thread for one of its high priority maintenance tasks that operates by checking some state of that task at a set interval to ensure it has made progress. The key challenge here is that the task is only expected to make progress when the application it belongs to has the opportunity to run, but there's no application-relative time to set the watchdog timer against. Instead, the next timeout is set relative to system time, using an approximation that assumes the application will continue to be scheduled as normal. If the task misses that approximate deadline because the application was frozen, without any way to know that, the watchdog may kill the healthy process.
Other sources of delay can cause similar issues, but this change focuses on allowing frozen time to be accounted for in particular because of how large it can grow and how unevenly it can affect applications running on the system. To allow an application to better account for the time it spends running, I propose tracking the time each cgroup spends freezing and exposing it to userland via a new core interface file in cgroupfs (cgroup.freeze.stat). I used this naming because utility controllers like "kill" and "freeze" are exposed as cgroup v2 core interface files, but I'm happy to change it if there's a convention others would prefer!
Currently, the cgroup css_set_lock is used to serialize accesses to the CGRP_FREEZE bit of cgrp->flags and the new cgroup_freezer_state counters (freeze_time_start_ns and freeze_time_total_ns). If we start to see higher contention on this lock, we may want to introduce a v2 freezer state-specific lock to avoid having to take the global lock every time a cgroup.freeze.stat file is read.
Any feedback would be much appreciated!
Thank you,
Tiffany
---
v2:
* Track per-cgroup freezing time instead of per-task frozen time as suggested by Tejun Heo
--- Documentation/admin-guide/cgroup-v2.rst | 8 ++++++++ include/linux/cgroup-defs.h | 6 ++++++ kernel/cgroup/cgroup.c | 24 ++++++++++++++++++++++++ kernel/cgroup/freezer.c | 8 ++++++-- 4 files changed, 44 insertions(+), 2 deletions(-)
Documentation/admin-guide/cgroup-v2.rst +8 -0
@@ -1018,6 +1018,14 @@ All cgroup core files are prefixed with "cgroup." it's possible to delete a frozen (and empty) cgroup, as well as create new sub-cgroups. + cgroup.freeze.stat + A read-only flat-keyed file which exists in non-root cgroups. + The following entry is defined: + + freeze_time_total_ns + Cumulative time that this cgroup has spent in the freezing + state, regardless of whether or not it reaches "frozen". + cgroup.kill A write-only single value file which exists in non-root cgroups. The only allowed value is "1".
include/linux/cgroup-defs.h +6 -0
@@ -436,6 +436,12 @@ struct cgroup_freezer_state {* frozen, SIGSTOPped, and PTRACEd. */ int nr_frozen_tasks; + +/* Time when the cgroup was requested to freeze */ +u64 freeze_time_start_ns; + +/* Total duration the cgroup has spent freezing */ +u64 freeze_time_total_ns; };struct cgroup {
kernel/cgroup/cgroup.c +24 -0
@@ -4050,6 +4050,23 @@ static ssize_t cgroup_freeze_write(struct kernfs_open_file *of,return nbytes; } +static int cgroup_freeze_stat_show (struct seq_file *seq,void *v) +{ +struct cgroup *cgrp =seq_css (seq)->cgroup; +u64 freeze_time =0 ; + +spin_lock_irq (&css_set_lock); +if (test_bit (CGRP_FREEZE , &cgrp->flags)) + freeze_time =ktime_get_ns () - cgrp->freezer.freeze_time_start_ns; + + freeze_time += cgrp->freezer.freeze_time_total_ns; +spin_unlock_irq (&css_set_lock); + +seq_printf (seq,"freeze_time_total_ns %llu\n" , freeze_time); + +return 0 ; +} +static void __cgroup_kill (struct cgroup *cgrp) {struct css_task_iter it;
@@ -5355,6 +5372,11 @@ static struct cftype cgroup_base_files[] = { .seq_show = cgroup_freeze_show, .write = cgroup_freeze_write, }, + { + .name = "cgroup.freeze.stat", + .flags = CFTYPE_NOT_ON_ROOT, + .seq_show = cgroup_freeze_stat_show, + }, { .name = "cgroup.kill", .flags = CFTYPE_NOT_ON_ROOT,
@@ -5758,6 +5780,7 @@ static struct cgroup *cgroup_create(struct cgroup *parent, const char *name,* if the parent has to be frozen, the child has too. */ cgrp->freezer.e_freeze = parent->freezer.e_freeze; + cgrp->freezer.freeze_time_total_ns =0 ;if (cgrp->freezer.e_freeze) {/* * Set the CGRP_FREEZE flag, so when a process will be
@@ -5766,6 +5789,7 @@ static struct cgroup *cgroup_create(struct cgroup *parent, const char *name,* consider it frozen immediately. */ set_bit (CGRP_FREEZE , &cgrp->flags); + cgrp->freezer.freeze_time_start_ns =ktime_get_ns ();set_bit (CGRP_FROZEN , &cgrp->flags); }
kernel/cgroup/freezer.c +6 -2
@@ -179,10 +179,14 @@ static void cgroup_do_freeze(struct cgroup *cgrp, bool freeze)lockdep_assert_held (&cgroup_mutex);spin_lock_irq (&css_set_lock); -if (freeze) +if (freeze) {set_bit (CGRP_FREEZE , &cgrp->flags); -else + cgrp->freezer.freeze_time_start_ns =ktime_get_ns (); + }else {clear_bit (CGRP_FREEZE , &cgrp->flags); + cgrp->freezer.freeze_time_total_ns += (ktime_get_ns () - + cgrp->freezer.freeze_time_start_ns); + }spin_unlock_irq (&css_set_lock);if (freeze)
Signature · 1 line
2.50.0.727.gbf7dc18ff4-goog
Michal Koutný
maintainerHello Tiffany.
Quoted 4 lines · Tiffany Yang
Other sources of delay can cause similar issues, but this change focuses on allowing frozen time to be accounted for in particular because of how large it can grow and how unevenly it can affect applications running on the system.
I'd like to incorporate the reason from your other mail:
| Since there isn't yet a clear way to identify a set of "lost" time | that everyone (or at least a wider group of users) cares about, it | seems like iterating over components of interest is the best way
into this commit message (because that's a stronger ponit that your use case alone).
Quoted 1 line
Any feedback would be much appreciated!
I can see benefits of this new stat field conceptually, I have some remarks to implementation and suggestions to conventions below.
Documentation/admin-guide/cgroup-v2.rst All cgroup core files are prefixed with "cgroup." · line 1021
Quoted 7 lines
Documentation/admin-guide/cgroup-v2.rst +1
-0@@ -1018,6 +1018,14 @@ All cgroup core files are prefixed with "cgroup." it's possible to delete a frozen (and empty) cgroup, as well as create new sub-cgroups. + cgroup.freeze.stat
With the given implementation (and use scenario), this'd better exposed in
cgroup.freeze.stat.local
I grok the hierarchical summing would make little sense and it'd make implementaion more complex. With that I'm thinking about formulation:
Cumulative time that cgroup has spent between freezing and thawing, regardless of whether by self or ancestor cgroups. NB (not) reaching "frozen" state is not accounted here.
Documentation/admin-guide/cgroup-v2.rst All cgroup core files are prefixed with "cgroup." · line 1028
Quoted 7 lines
Documentation/admin-guide/cgroup-v2.rst +7
-0+ A read-only flat-keyed file which exists in non-root cgroups. + The following entry is defined: + + freeze_time_total_ns + Cumulative time that this cgroup has spent in the freezing + state, regardless of whether or not it reaches "frozen". +
Rather use microseconds, it's the cgroup API convention and I'm not sure nanosecods exposed here are the needed precision.
1 _____
frozen 0 __/ \__
ab cd
Yeah, I find the mesurent between a and c the sanest.
Documentation/admin-guide/cgroup-v2.rst All cgroup core files are prefixed with "cgroup."
Quoted 11 lines
Documentation/admin-guide/cgroup-v2.rst +11
-0+static int cgroup_freeze_stat_show(struct seq_file *seq, void *v) +{ + struct cgroup *cgrp = seq_css(seq)->cgroup; + u64 freeze_time = 0; + + spin_lock_irq(&css_set_lock); + if (test_bit(CGRP_FREEZE, &cgrp->flags)) + freeze_time = ktime_get_ns() - cgrp->freezer.freeze_time_start_ns; + + freeze_time += cgrp->freezer.freeze_time_total_ns; + spin_unlock_irq(&css_set_lock);
I don't like taking this spinlock only for the matter of reading this attribute. The intention should be to keep the (un)freezeing mostly unaffected at the expense of these readers (seqcount or u64 stats?).
Alternative approach: either there's outer watcher who can be notified by cgroup.events:frozen or it's an inner watcher who couldn't actively read the field anyway. So the field could only show completed freeze/thaw cycles from the past (i.e. not substitute clock_gettime(2) when the cgroup is frozen), which could simplify querying the flag too.
Documentation/admin-guide/cgroup-v2.rst static struct cgroup *cgroup_create(struct cgroup *parent, const char *name, · line 5783
Quoted 5 lines
Documentation/admin-guide/cgroup-v2.rst +1
-0@@ -5758,6 +5780,7 @@ static struct cgroup *cgroup_create(struct cgroup *parent, const char *name, * if the parent has to be frozen, the child has too. */ cgrp->freezer.e_freeze = parent->freezer.e_freeze; + cgrp->freezer.freeze_time_total_ns = 0;
struct cgroup is kzalloc'd, this is unnecessary
Quoted 28 lines · Michal Koutný
Hello Tiffany.
Quoted 4 lines · Tiffany Yang
Other sources of delay can cause similar issues, but this change focuses on allowing frozen time to be accounted for in particular because of how large it can grow and how unevenly it can affect applications running on the system.
I'd like to incorporate the reason from your other mail:
| Since there isn't yet a clear way to identify a set of "lost" time | that everyone (or at least a wider group of users) cares about, it | seems like iterating over components of interest is the best wayinto this commit message (because that's a stronger ponit that your use case alone).
Quoted 1 line
Any feedback would be much appreciated!
I can see benefits of this new stat field conceptually, I have some remarks to implementation and suggestions to conventions below.
Quoted 6 lines
Documentation/admin-guide/cgroup-v2.rst +0
-0@@ -1018,6 +1018,14 @@ All cgroup core files are prefixed with "cgroup." it's possible to delete a frozen (and empty) cgroup, as well as create new sub-cgroups.
Quoted 6 lines
Documentation/admin-guide/cgroup-v2.rst All cgroup core files are prefixed with "cgroup." · line 1021
Quoted 1 line
Documentation/admin-guide/cgroup-v2.rst +1
-0+ cgroup.freeze.statWith the given implementation (and use scenario), this'd better exposed in
cgroup.freeze.stat.local
Would it be possible to add this field to either cgroup.event or cgroup.stat? Since the frozen status is already tracked in cgroup.event, this placement would maintain better cohesion with existing metrics.
This is just a suggestion.
Best regards,
Ridong
Quoted 55 lines
I grok the hierarchical summing would make little sense and it'd make implementaion more complex. With that I'm thinking about formulation:
Cumulative time that cgroup has spent between freezing and thawing, regardless of whether by self or ancestor cgroups. NB (not) reaching "frozen" state is not accounted here.
Documentation/admin-guide/cgroup-v2.rst All cgroup core files are prefixed with "cgroup." · line 1028
Quoted 7 lines
Documentation/admin-guide/cgroup-v2.rst +7
-0+ A read-only flat-keyed file which exists in non-root cgroups. + The following entry is defined: + + freeze_time_total_ns + Cumulative time that this cgroup has spent in the freezing + state, regardless of whether or not it reaches "frozen". +Rather use microseconds, it's the cgroup API convention and I'm not sure nanosecods exposed here are the needed precision.
1 _____frozen 0 __/ \__
ab cdYeah, I find the mesurent between a and c the sanest.
Documentation/admin-guide/cgroup-v2.rst All cgroup core files are prefixed with "cgroup."
Quoted 11 lines
Documentation/admin-guide/cgroup-v2.rst +11
-0+static int cgroup_freeze_stat_show(struct seq_file *seq, void *v) +{ + struct cgroup *cgrp = seq_css(seq)->cgroup; + u64 freeze_time = 0; + + spin_lock_irq(&css_set_lock); + if (test_bit(CGRP_FREEZE, &cgrp->flags)) + freeze_time = ktime_get_ns() - cgrp->freezer.freeze_time_start_ns; + + freeze_time += cgrp->freezer.freeze_time_total_ns; + spin_unlock_irq(&css_set_lock);I don't like taking this spinlock only for the matter of reading this attribute. The intention should be to keep the (un)freezeing mostly unaffected at the expense of these readers (seqcount or u64 stats?).
Alternative approach: either there's outer watcher who can be notified by cgroup.events:frozen or it's an inner watcher who couldn't actively read the field anyway. So the field could only show completed freeze/thaw cycles from the past (i.e. not substitute
clock_gettime(2)when the cgroup is frozen), which could simplify querying the flag too.Documentation/admin-guide/cgroup-v2.rst static struct cgroup *cgroup_create(struct cgroup *parent, const char *name, · line 5783
Quoted 5 lines
Documentation/admin-guide/cgroup-v2.rst +1
-0@@ -5758,6 +5780,7 @@ static struct cgroup *cgroup_create(struct cgroup *parent, const char *name, * if the parent has to be frozen, the child has too. */ cgrp->freezer.e_freeze = parent->freezer.e_freeze; + cgrp->freezer.freeze_time_total_ns = 0;struct cgroup is kzalloc'd, this is unnecessary
Tejun Heo
maintainerQuoted 10 lines · Chen Ridong
Quoted 4 lines
With the given implementation (and use scenario), this'd better exposed in
cgroup.freeze.stat.localWould it be possible to add this field to either cgroup.event or cgroup.stat? Since the frozen status is already tracked in cgroup.event, this placement would maintain better cohesion with existing metrics.
This is just a suggestion.
Yeah, given that the freezer is an integral part of cgroup core, using cgroup.stat[.local] probably makes more sense.
Thanks.
Signature · 1 line
tejun
Quoted 10 lines · Tejun Heo
Quoted 8 lines · Chen Ridong
Quoted 4 lines
With the given implementation (and use scenario), this'd better exposed in
cgroup.freeze.stat.localWould it be possible to add this field to either cgroup.event or cgroup.stat? Since the frozen status is already tracked in cgroup.event, this placement would maintain better cohesion with existing metrics.
Quoted 6 lines
Quoted 1 line
This is just a suggestion.
Yeah, given that the freezer is an integral part of cgroup core, using cgroup.stat[.local] probably makes more sense.
Thanks.
One of the reasons I avoided cgroup.stat was because I interpreted its purpose to be for exposing cgroup metadata (i.e., descendants and descendants per subsystem), and I didn't think this value fit in neatly.
It doesn't seem like there currently exists a cgroup.stat.local, but if that is the preferred location for this accounting, I could create one and print it there!
Signature · 1 line
Tiffany Y. Yang
Hi Rindong,
Quoted 18 lines · Chen Ridong
Quoted 10 lines · Michal Koutný
Hello Tiffany.
Quoted 5 lines · Tiffany Yang
Documentation/admin-guide/cgroup-v2.rst +0
-0@@ -1018,6 +1018,14 @@ All cgroup core files are prefixed with "cgroup." it's possible to delete a frozen (and empty) cgroup, as well as create new sub-cgroups.Quoted 5 lines
Documentation/admin-guide/cgroup-v2.rst All cgroup core files are prefixed with "cgroup." · line 1021
Quoted 1 line
Documentation/admin-guide/cgroup-v2.rst +1
-0+ cgroup.freeze.statWith the given implementation (and use scenario), this'd better exposed in
cgroup.freeze.stat.local
Quoted 8 lines
Would it be possible to add this field to either cgroup.event or cgroup.stat? Since the frozen status is already tracked in cgroup.event, this placement would maintain better cohesion with existing metrics.
This is just a suggestion.
Best regards,
Ridong
Thanks for taking a look!
I don't think this would *quite* fit in cgroup.event because we're measuring when the cgroup begins freezing instead of when it reaches the frozen state. I also worry that having the value so close to cgroup.frozen would cause additional confusion about its meaning. cgroup.stat seems reasonable, but the values inside appear to be accounted for hierarchically, which wouldn't suit our use case.
Signature · 1 line
Tiffany Y. Yang
Tejun Heo
maintainerHello,
On Thu, Jul 17, 2025 at 02:56:13PM +0200, Michal Koutný wrote:
...
Quoted 6 lines
Quoted 1 line
+ cgroup.freeze.stat
With the given implementation (and use scenario), this'd better exposed in
cgroup.freeze.stat.local
Quoted 6 lines
I grok the hierarchical summing would make little sense and it'd make implementaion more complex. With that I'm thinking about formulation:
Cumulative time that cgroup has spent between freezing and thawing, regardless of whether by self or ancestor cgroups. NB (not) reaching "frozen" state is not accounted here.
I wonder what hierarchical summing would look like for this. It's absolute time interval measurement and I'm not sure whether summing up the descendants' durations is the best way to go about it. ie. Should it be the total duration any of the descendants are freezing or should it be sum of the freezing durations of all descendants?
Thanks.
Signature · 1 line
tejun
Michal Koutný
maintainerQuoted 1 line · Tejun Heo
I wonder what hierarchical summing would look like for this.
So do I.
Thus I meant to expose this only in a *.local file not the hierarchical one.
But I realize it should [1] match cpu.stat[.local]:thottled_usec since they're similar quantities in principle.
- cpu.stat:thottled_usec
- sums the time the cgroup's quota was in effect
- not hierarchical (:-/)
- cpu.stat.local:thottled_usec
- not hierarchical
- sums the time cgroup's or ancestor's quota was in effect
-> IIUC this is what's the motivation of the original patch
HTH,
Michal
[1] I'd find it more logical if
cpu.stat:thottled_usec were cpu.stat.local:thottling_usec and cpu.stat.local:thottled_usec were cpu.stat.local:throttled_usec. Only to illustrate my understanding of hierarchy in cpu.stat, it doesn't matter since it's what it is now.
Quoted 17 lines · Michal Koutný
Quoted 1 line · Tejun Heo
I wonder what hierarchical summing would look like for this.
So do I.
Thus I meant to expose this only in a *.local file not the hierarchical one.But I realize it should [1] match cpu.stat[.local]:thottled_usec since they're similar quantities in principle.
- cpu.stat:thottled_usec
- sums the time the cgroup's quota was in effect
- not hierarchical (:-/)
- cpu.stat.local:thottled_usec
- not hierarchical
- sums the time cgroup's or ancestor's quota was in effect-> IIUC this is what's the motivation of the original patch
Quoted 8 lines
HTH,
Michal[1] I'd find it more logical if
cpu.stat:thottled_usec were cpu.stat.local:thottling_usec and cpu.stat.local:thottled_usec were cpu.stat.local:throttled_usec. Only to illustrate my understanding of hierarchy in cpu.stat, it doesn't matter since it's what it is now.
Hi Michal and TJ,
I'd like to raise a separate thought unrelated to the current discussion.:)
With the recent merge of the series "cgroup: separate rstat trees," the rstat are not bound to CPU system. This makes me wonder: should we consider moving the cpu.stat and cpu.stat.local interfaces to the CPU subsystem?
The CPU subsystem could then align more closely with other resource controllers like memory or I/O subsystems. By decoupling these CPU-specific statistics from the cgroup core, it could help keep both cgroup and rstat implementations more focused.
Is there any particular reason why the CPU subsystem must remain bound to the cgroup core?
Looking forward to your insights.
Best regards,
Ridong
Michal Koutný
maintainercpu.stat in core or cpu controller (was Re: [RFC PATCH v2] cgroup: Track time in cgroup v2 freezer)
Quoted 3 lines · Chen Ridong
With the recent merge of the series "cgroup: separate rstat trees," the rstat are not bound to CPU system. This makes me wonder: should we consider moving the cpu.stat and cpu.stat.local interfaces to the CPU subsystem?
Note that fields printed in cpu.stat are combination of "core" and cpu controller values.
Quoted 3 lines
The CPU subsystem could then align more closely with other resource controllers like memory or I/O subsystems. By decoupling these CPU-specific statistics from the cgroup core, it could help keep both cgroup and rstat implementations more focused.
In my eyes, cpu controller is stuff encapsulated by cpu_cgrp_subsys. I'm not sure I understand what you refer to as the CPU subsystem.
One thing is how it is presented to users (filenames and content) another one is how it is implemented. The latter surely can be refactored but it's not obvious to me from the short description, sorry.
Quoted 2 lines
Is there any particular reason why the CPU subsystem must remain bound to the cgroup core?
The stuff that's bound to the core is essentially not "control" but only accounting, so with this association, the accounting can have fine granularity while control (which incurs higher overhead in principle) may remain coarse. I find it thus quite fitting that CPU stats build on top of rstat.
(Naturally, my previous claim about overhead is only rough and it's the reason for existence of adjustments like in the commit 34f26a15611af ("sched/psi: Per-cgroup PSI accounting disable/re-enable interface").)
Thats how I see it, happy to discuss possible problems you see with this.
Michal
Re: cpu.stat in core or cpu controller (was Re: [RFC PATCH v2] cgroup: Track time in cgroup v2 freezer)
Quoted 8 lines · Michal Koutný
Quoted 3 lines · Chen Ridong
With the recent merge of the series "cgroup: separate rstat trees," the rstat are not bound to CPU system. This makes me wonder: should we consider moving the cpu.stat and cpu.stat.local interfaces to the CPU subsystem?
Note that fields printed in cpu.stat are combination of "core" and cpu controller values.
Do you mean the "core" values are shown as below:
- usage_usec
- user_usec
- system_usec
- nice_usec
In the legacy cgroup, these values are in the cpuacct subsystem.
Quoted 11 lines
Quoted 3 lines
The CPU subsystem could then align more closely with other resource controllers like memory or I/O subsystems. By decoupling these CPU-specific statistics from the cgroup core, it could help keep both cgroup and rstat implementations more focused.
In my eyes, cpu controller is stuff encapsulated by
cpu_cgrp_subsys. I'm not sure I understand what you refer to as the CPU subsystem.One thing is how it is presented to users (filenames and content) another one is how it is implemented. The latter surely can be refactored but it's not obvious to me from the short description, sorry.
What I'm considering is moving the implementation of cpu.stat from cgroup_base_files to cpu_cgrp_subsys—without changing the user-facing interface (filenames and content remain the same). However, the interface would only appear if the CPU subsystem is enabled.
Currently, cpu.stat and cpu.stat.local are visible in every cgroup, even when the CPU subsystem is disabled. The only populated fields in such cases are:
- usage_usec
- user_usec
- system_usec
- nice_usec
I’m unsure whether this change would be acceptable?
Quoted 8 lines
Quoted 2 lines
Is there any particular reason why the CPU subsystem must remain bound to the cgroup core?
The stuff that's bound to the core is essentially not "control" but only accounting, so with this association, the accounting can have fine granularity while control (which incurs higher overhead in principle) may remain coarse. I find it thus quite fitting that CPU stats build on top of rstat.
The implementation would still rely on rstat, similar to memory.stat and io.stat. The goal is to decouple it from the cgroup core (cgroup.c and rstat.c) while preserving accounting granularity.
Best regards,
Ridong
Quoted 8 lines
(Naturally, my previous claim about overhead is only rough and it's the reason for existence of adjustments like in the commit
34f26a15611af("sched/psi: Per-cgroup PSI accounting disable/re-enable interface").)Thats how I see it, happy to discuss possible problems you see with this.
Michal
Tejun Heo
maintainerRe: cpu.stat in core or cpu controller (was Re: [RFC PATCH v2] cgroup: Track time in cgroup v2 freezer)
On Sat, Jul 19, 2025 at 10:01:07AM +0800, Chen Ridong wrote:
...
Quoted 7 lines
What I'm considering is moving the implementation of cpu.stat from
cgroup_base_filesto cpu_cgrp_subsys—without changing the user-facing interface (filenames and content remain the same). However, the interface would only appear if the CPU subsystem is enabled.Currently, cpu.stat and cpu.stat.local are visible in every cgroup, even when the CPU subsystem is disabled. The only populated fields in such cases are:
Quoted 6 lines
-
usage_usec
-user_usec
-system_usec
-nice_usecI’m unsure whether this change would be acceptable?
I don't think so and don't really see what benefits moving the stats would bring. Why would we move these?
Thanks.
Signature · 1 line
tejun
Re: cpu.stat in core or cpu controller (was Re: [RFC PATCH v2] cgroup: Track time in cgroup v2 freezer)
Quoted 14 lines · Tejun Heo
On Sat, Jul 19, 2025 at 10:01:07AM +0800, Chen Ridong wrote:
...Quoted 12 lines
What I'm considering is moving the implementation of cpu.stat from
cgroup_base_filesto cpu_cgrp_subsys—without changing the user-facing interface (filenames and content remain the same). However, the interface would only appear if the CPU subsystem is enabled.Currently, cpu.stat and cpu.stat.local are visible in every cgroup, even when the CPU subsystem is disabled. The only populated fields in such cases are:
-
usage_usec
-user_usec
-system_usec
-nice_usec
Quoted 7 lines
Quoted 1 line
I’m unsure whether this change would be acceptable?
I don't think so and don't really see what benefits moving the stats would bring. Why would we move these?
Thanks.
Thank you for your attention. My intention is to better modularize the cgroup code by moving CPU subsystem-specific statistics out of the core cgroup implementation (cgroup.c and rstat.c).
Specifically, this change would allow us to:
1.Remove these CPU-specific callbacks from the core:
css_extra_stat_show()
css_local_stat_show()
2. Clean up the 'is_self' logic in rstat.c.
3. Make the stat handling consistent across subsystems (currently cpu.stat is the only subsystem-specific stat implemented in the core).
Best regards,
Ridong.
Michal Koutný
maintainerRe: cpu.stat in core or cpu controller (was Re: [RFC PATCH v2] cgroup: Track time in cgroup v2 freezer)
Quoted 6 lines · Chen Ridong
Specifically, this change would allow us to:
1.Remove these CPU-specific callbacks from the core:
css_extra_stat_show()css_local_stat_show()
2. Clean up the 'is_self' logic inrstat.c.
If you see an option to organize the code better, why not. (At the same time, I currently also don't see the "why.)
Quoted 2 lines
3. Make the stat handling consistent across subsystems (currently cpu.stat is the only subsystem-specific stat implemented in the core).
But beware that the possibility of having cpu.stat without enabling the cpu controller on v2 is a user visible behavior and I'm quite sure some userspace relies on it, so you'd need to preserve that.
Michal
Re: cpu.stat in core or cpu controller (was Re: [RFC PATCH v2] cgroup: Track time in cgroup v2 freezer)
Quoted 12 lines · Michal Koutný
Quoted 6 lines · Chen Ridong
Specifically, this change would allow us to:
1.Remove these CPU-specific callbacks from the core:
css_extra_stat_show()css_local_stat_show()
2. Clean up the 'is_self' logic inrstat.c.If you see an option to organize the code better, why not. (At the same time, I currently also don't see the "why.)
Quoted 7 lines
Quoted 2 lines
3. Make the stat handling consistent across subsystems (currently cpu.stat is the only subsystem-specific stat implemented in the core).
But beware that the possibility of having cpu.stat without enabling the cpu controller on v2 is a user visible behavior and I'm quite sure some userspace relies on it, so you'd need to preserve that.
This is what I worry about. Thank you for your confirmation.
Best regards,
Ridong
Tejun Heo
maintainerRe: cpu.stat in core or cpu controller (was Re: [RFC PATCH v2] cgroup: Track time in cgroup v2 freezer)
Quoted 5 lines · Chen Ridong
Quoted 3 lines
But beware that the possibility of having cpu.stat without enabling the cpu controller on v2 is a user visible behavior and I'm quite sure some userspace relies on it, so you'd need to preserve that.
This is what I worry about. Thank you for your confirmation.
Yeah, this was an intentional decision - sacrificing a bit of code org cleanliness for everyday usefulness. Enabling CPU controller can have substantial overhead and having cpu stats available by default doesn't cost much while improving usefulness.
Thanks.
Signature · 1 line
tejun
Re: cpu.stat in core or cpu controller (was Re: [RFC PATCH v2] cgroup: Track time in cgroup v2 freezer)
Quoted 5 lines · Tejun Heo
Quoted 4 lines · Chen Ridong
Quoted 3 lines
But beware that the possibility of having cpu.stat without enabling the cpu controller on v2 is a user visible behavior and I'm quite sure some userspace relies on it, so you'd need to preserve that.
Quoted 9 lines
Quoted 1 line
This is what I worry about. Thank you for your confirmation.
Yeah, this was an intentional decision - sacrificing a bit of code org cleanliness for everyday usefulness. Enabling CPU controller can have substantial overhead and having cpu stats available by default doesn't cost much while improving usefulness.
Thanks.
Thank you Tj, This is clear now.
Best regards,
Ridong
Quoted 32 lines · Michal Koutný
I'd like to incorporate the reason from your other mail:
| Since there isn't yet a clear way to identify a set of "lost" time | that everyone (or at least a wider group of users) cares about, it | seems like iterating over components of interest is the best wayinto this commit message (because that's a stronger ponit that your use case alone).
Quoted 1 line
Any feedback would be much appreciated!
I can see benefits of this new stat field conceptually, I have some remarks to implementation and suggestions to conventions below.
Documentation/admin-guide/cgroup-v2.rst All cgroup core files are prefixed with "cgroup." · line 1021
Quoted 7 lines
Documentation/admin-guide/cgroup-v2.rst +1
-0@@ -1018,6 +1018,14 @@ All cgroup core files are prefixed with "cgroup." it's possible to delete a frozen (and empty) cgroup, as well as create new sub-cgroups. + cgroup.freeze.statWith the given implementation (and use scenario), this'd better exposed in
cgroup.freeze.stat.localI grok the hierarchical summing would make little sense and it'd make implementaion more complex. With that I'm thinking about formulation:
Cumulative time that cgroup has spent between freezing and thawing, regardless of whether by self or ancestor cgroups. NB (not) reaching "frozen" state is not accounted here.
Quoted 10 lines
Documentation/admin-guide/cgroup-v2.rst All cgroup core files are prefixed with "cgroup." · line 1028
Quoted 7 lines
Documentation/admin-guide/cgroup-v2.rst +7
-0+ A read-only flat-keyed file which exists in non-root cgroups. + The following entry is defined: + + freeze_time_total_ns + Cumulative time that this cgroup has spent in the freezing + state, regardless of whether or not it reaches "frozen". +Rather use microseconds, it's the cgroup API convention and I'm not sure nanosecods exposed here are the needed precision.
Ack.
Quoted 7 lines
1 _____frozen 0 __/ \__
ab cdYeah, I find the mesurent between a and c the sanest.
Quoted 15 lines
Documentation/admin-guide/cgroup-v2.rst All cgroup core files are prefixed with "cgroup."
Quoted 11 lines
Documentation/admin-guide/cgroup-v2.rst +11
-0+static int cgroup_freeze_stat_show(struct seq_file *seq, void *v) +{ + struct cgroup *cgrp = seq_css(seq)->cgroup; + u64 freeze_time = 0; + + spin_lock_irq(&css_set_lock); + if (test_bit(CGRP_FREEZE, &cgrp->flags)) + freeze_time = ktime_get_ns() - cgrp->freezer.freeze_time_start_ns; + + freeze_time += cgrp->freezer.freeze_time_total_ns; + spin_unlock_irq(&css_set_lock);I don't like taking this spinlock only for the matter of reading this attribute. The intention should be to keep the (un)freezeing mostly unaffected at the expense of these readers (seqcount or u64 stats?).
Ah, thank you for this suggestion! I noticed that none of the other seq_file read implementations took a lock, so I thought this might be a point of contention. I'll try a seqlock in the next version of the patch.
Quoted 5 lines
Alternative approach: either there's outer watcher who can be notified by cgroup.events:frozen or it's an inner watcher who couldn't actively read the field anyway. So the field could only show completed freeze/thaw cycles from the past (i.e. not substitute
clock_gettime(2)when the cgroup is frozen), which could simplify querying the flag too.
This is a good observation. This approach does simplify things, but even though it would work for our use case, I feel like this value would be less useful for the outer watcher case, especially in the case where the cgroup never reaches the frozen state.
Quoted 8 lines
Documentation/admin-guide/cgroup-v2.rst static struct cgroup *cgroup_create(struct cgroup · line 5784
Quoted 6 lines
Documentation/admin-guide/cgroup-v2.rst +1
-0@@ -5758,6 +5780,7 @@ static struct cgroup *cgroup_create(struct cgroup *parent, const char *name, * if the parent has to be frozen, the child has too. */ cgrp->freezer.e_freeze = parent->freezer.e_freeze; + cgrp->freezer.freeze_time_total_ns = 0;struct cgroup is kzalloc'd, this is unnecessary
Thank you for all your feedback! I'll make sure to incorporate these suggestions into the next version.
Signature · 1 line
Tiffany Y. Yang