freezer for cgroup v2

Now the number of descendant cgroups and the number of dying descendant cgroups are synchronized using the cgroup_mutex.

The number of descendant cgroups will be required by the cgroup v2 freezer, which will use it to determine if a cgroup is frozen (depending on total number of descendants and number of frozen descendants). It's not always acceptable to grab the cgroup_mutex, especially from quite hot paths (e.g. exit()).

To avoid this, let's additionally synchronize these counters using the css_set_lock.

kernel/cgroup/freezer.c new file +24 -0

+void cgroup_freezer_migrate_task(struct task_struct *task,
+				 struct cgroup *src, struct cgroup *dst)
+{
+	lockdep_assert_held(&css_set_lock);
+
+	/*
+	 * Kernel threads are not supposed to be frozen at all.
+	 */
+	if (task->flags & PF_KTHREAD)
+		return;
+
+	/*
+	 * Adjust counters of freezing and frozen tasks.
+	 */
+	if (test_bit(CGRP_FREEZE, &src->flags)) {
+		src->freezer.nr_tasks_to_freeze--;
+		WARN_ON_ONCE(src->freezer.nr_tasks_to_freeze < 0);
+	}
+
+	/*
+	 * If the task is frozen, let's bump nr_tasks_to_freeze even
+	 * if the target cgroup isn't frozen: the counter will be decreased
+	 * in cgroup_leave_frozen().
+	 */

Quoted 29 lines · Roman Gushchin

kernel/cgroup/freezer.c new file +29 -0

+void cgroup_freezer_migrate_task(struct task_struct *task,
+				 struct cgroup *src, struct cgroup *dst)
+{
+	lockdep_assert_held(&css_set_lock);
+
+	/*
+	 * Kernel threads are not supposed to be frozen at all.
+	 */
+	if (task->flags & PF_KTHREAD)
+		return;
+
+	/*
+	 * Adjust counters of freezing and frozen tasks.
+	 */
+	if (test_bit(CGRP_FREEZE, &src->flags)) {
+		src->freezer.nr_tasks_to_freeze--;
+		WARN_ON_ONCE(src->freezer.nr_tasks_to_freeze < 0);
+	}
+
+	/*
+	 * If the task is frozen, let's bump nr_tasks_to_freeze even
+	 * if the target cgroup isn't frozen: the counter will be decreased
+	 * in cgroup_leave_frozen().
+	 */
+	if (test_bit(CGRP_FREEZE, &dst->flags) || task->frozen)
+		dst->freezer.nr_tasks_to_freeze++;
+
+	if (task->frozen) {
+		src->freezer.nr_frozen_tasks--;

kernel/cgroup/freezer.c line 233

Quoted 5 lines

kernel/cgroup/freezer.c new file +5 -0

+		dst->freezer.nr_frozen_tasks++;
+		WARN_ON_ONCE(src->freezer.nr_frozen_tasks < 0);
+		WARN_ON_ONCE(dst->freezer.nr_frozen_tasks >
+			     dst->freezer.nr_tasks_to_freeze);
+	}

If a non-frozen task is being moved into a frozen cgroup, shouldn't that also trigger frozen state update?

Quoted 2 lines

If a non-frozen task is being moved into a frozen cgroup, shouldn't that also trigger frozen state update?

It does! Just below these lines:

	/*
	 * If the task isn't in the desired state, force it to it.
	 */
	if (task->frozen != test_bit(CGRP_FREEZE, &dst->flags))
		cgroup_freeze_task(task, test_bit(CGRP_FREEZE, &dst->flags));

Quoted 5 lines · Roman Gushchin

Quoted 2 lines

If a non-frozen task is being moved into a frozen cgroup, shouldn't that also trigger frozen state update?

It does! Just below these lines:

Quoted 5 lines

	/*
	 * If the task isn't in the desired state, force it to it.
	 */
	if (task->frozen != test_bit(CGRP_FREEZE, &dst->flags))
		cgroup_freeze_task(task, test_bit(CGRP_FREEZE, &dst->flags));

But that wouldn't udpate the cgroup's frozen state and generate notifications, right?

Quoted 2 lines

But that wouldn't udpate the cgroup's frozen state and generate notifications, right?

Why? The task will be eventually trapped into cgroup_enter_frozen(), and from there cgroup_update_frozen() will be called.

You are right, that notification will not be issued, because the cgroup is not changing its state (frozen->frozen). I'm not sure that it makes sense to change the cgroup state back and forth in this case. Are there any reasons I'm missing?

Hello,

Quoted 5 lines · Roman Gushchin

Quoted 2 lines

But that wouldn't udpate the cgroup's frozen state and generate notifications, right?

Why? The task will be eventually trapped into cgroup_enter_frozen(), and from there cgroup_update_frozen() will be called.

Because the cgroup is no longer frozen?

Quoted 4 lines

You are right, that notification will not be issued, because the cgroup is not changing its state (frozen->frozen). I'm not sure that it makes sense to change the cgroup state back and forth in this case. Are there any reasons I'm missing?

Imagine the task being trapped in nfs or wherever and not getting into the freezer for an extended period of time.

Yeah, it's a good point. I've thought about it mostly in the fork() context, where if freezing of a cgroup races with fork(), it makes no sense to switch the cgroup state back and forth. But that case is different, as the child will be trapped just on the return path from fork() call.

Describe cgroup v2 freezer interface in the cgroup v2 admin guide.

Signed-off-by Roman Gushchin <[email protected]>

Cc Tejun Heo <[email protected]>

Cc [email protected]

Cc [email protected]

---
 Documentation/admin-guide/cgroup-v2.rst | 26 +++++++++++++++++++++++++
 1 file changed, 26 insertions(+)

Documentation/admin-guide/cgroup-v2.rst +2 -0

index 184193bcb262..a065c0bed88c 100644
@@ -862,6 +862,8 @@ All cgroup core files are prefixed with "cgroup."
 	  populated
 		1 if the cgroup or its descendants contains any live
 		processes; otherwise, 0.
+	  frozen
+		1 if the cgroup is frozen; otherwise, 0.
 
   cgroup.max.descendants
 	A read-write single value files.  The default is "max".

@@ -895,6 +897,30 @@ All cgroup core files are prefixed with "cgroup."
 		A dying cgroup can consume system resources not exceeding

Documentation/admin-guide/cgroup-v2.rst +0 -0

 Controllers
 ===========

Signature · 1 line

2.17.2

Hi,

Quoted 25 lines · Roman Gushchin

Describe cgroup v2 freezer interface in the cgroup v2 admin guide.

Signed-off-by Roman Gushchin <[email protected]>

Cc Tejun Heo <[email protected]>

Cc [email protected]

Cc [email protected]

---
 Documentation/admin-guide/cgroup-v2.rst | 26 +++++++++++++++++++++++++
 1 file changed, 26 insertions(+)

Documentation/admin-guide/cgroup-v2.rst +2 -0

index 184193bcb262..a065c0bed88c 100644
@@ -862,6 +862,8 @@ All cgroup core files are prefixed with "cgroup."
 	  populated
 		1 if the cgroup or its descendants contains any live
 		processes; otherwise, 0.
+	  frozen
+		1 if the cgroup is frozen; otherwise, 0.
 
   cgroup.max.descendants
 	A read-write single value files.  The default is "max".

@@ -895,6 +897,30 @@ All cgroup core files are prefixed with "cgroup."
 		A dying cgroup can consume system resources not exceeding

Documentation/admin-guide/cgroup-v2.rst All cgroup core files are prefixed with "cgroup." · line 907

Quoted 10 lines

Documentation/admin-guide/cgroup-v2.rst +8 -0

 		limits, which were active at the moment of cgroup deletion.
 
+  cgroup.freeze
+	A read-write single value file which exists on non-root cgroups.
+	Allowed values are "0" and "1". The default is "0".
+
+	Writing "1" to the file causes freezing of the cgroup and all
+	descendant cgroups. This means that all belonging processes will
+	be stopped and will not run until the cgroup will be explicitly
+	unfrozen. Freezing of the cgroup may take some time; when the process

"when the process is complete" sounds somewhat ambiguous, it's unclear whether freezing is complete or the process that's being frozen is complete.

Maybe "when this action is completed"?

Documentation/admin-guide/cgroup-v2.rst All cgroup core files are prefixed with "cgroup." · line 909

Quoted 2 lines

Documentation/admin-guide/cgroup-v2.rst +2 -0

+	is complete, the "frozen" value in the cgroup.events control file
+	will be updated and the corresponding notification will be issued.

Can you please clarify how exactly cgroup.events would be updated?

Documentation/admin-guide/cgroup-v2.rst All cgroup core files are prefixed with "cgroup." · line 911

Quoted 1 line

Documentation/admin-guide/cgroup-v2.rst +1 -0

+	Cgroup can be frozen either by its own settings, either by settings

^ A cgroup ... and maybe there are more "a" and "the" that should be
fixed, it's hard for me to tell.

Also, I believe "either ..., or ..." sounds better than "either ..., either ..."

Documentation/admin-guide/cgroup-v2.rst All cgroup core files are prefixed with "cgroup." · line 917

Quoted 6 lines

Documentation/admin-guide/cgroup-v2.rst +6 -0

+	of any ancestor cgroups. If any of ancestor cgroups is frozen, the
+	cgroup will remain frozen.
+
+	Processes in the frozen cgroup can be killed by a fatal signal.
+	They also can enter and leave a frozen cgroup: either by an explicit
+	move by a user, either if freezing of the cgroup races with fork().

ditto

Documentation/admin-guide/cgroup-v2.rst All cgroup core files are prefixed with "cgroup." · line 918

Quoted 1 line

Documentation/admin-guide/cgroup-v2.rst +1 -0

+	If a cgroup is moved to a frozen cgroup, it stops. If a process is

^ process?

Documentation/admin-guide/cgroup-v2.rst All cgroup core files are prefixed with "cgroup." · line 919

Quoted 1 line

Documentation/admin-guide/cgroup-v2.rst +1 -0

+	moving out of a frozen cgroup, it becomes running.

^ moved

Documentation/admin-guide/cgroup-v2.rst All cgroup core files are prefixed with "cgroup." · line 923

Quoted 4 lines

Documentation/admin-guide/cgroup-v2.rst +3 -0

+	Frozen status of a cgroup doesn't affect any cgroup tree operations:
+	it's possible to delete a frozen (and empty) cgroup, as well as
+	create new sub-cgroups.

Maybe it's also worth adding that freezing a process has no effect on its memory consumption, at least directly.

Quoted 67 lines · Mike Rapoport

Hi,

Quoted 25 lines · Roman Gushchin

Describe cgroup v2 freezer interface in the cgroup v2 admin guide.

Signed-off-by Roman Gushchin <[email protected]>

Cc Tejun Heo <[email protected]>

Cc [email protected]

Cc [email protected]

---
 Documentation/admin-guide/cgroup-v2.rst | 26 +++++++++++++++++++++++++
 1 file changed, 26 insertions(+)

Documentation/admin-guide/cgroup-v2.rst +2 -0

index 184193bcb262..a065c0bed88c 100644
@@ -862,6 +862,8 @@ All cgroup core files are prefixed with "cgroup."
 	  populated
 		1 if the cgroup or its descendants contains any live
 		processes; otherwise, 0.
+	  frozen
+		1 if the cgroup is frozen; otherwise, 0.
 
   cgroup.max.descendants
 	A read-write single value files.  The default is "max".

@@ -895,6 +897,30 @@ All cgroup core files are prefixed with "cgroup."
 		A dying cgroup can consume system resources not exceeding

Documentation/admin-guide/cgroup-v2.rst All cgroup core files are prefixed with "cgroup." · line 907

Quoted 10 lines

Documentation/admin-guide/cgroup-v2.rst +8 -0

 		limits, which were active at the moment of cgroup deletion.
 
+  cgroup.freeze
+	A read-write single value file which exists on non-root cgroups.
+	Allowed values are "0" and "1". The default is "0".
+
+	Writing "1" to the file causes freezing of the cgroup and all
+	descendant cgroups. This means that all belonging processes will
+	be stopped and will not run until the cgroup will be explicitly
+	unfrozen. Freezing of the cgroup may take some time; when the process

"when the process is complete" sounds somewhat ambiguous, it's unclear whether freezing is complete or the process that's being frozen is complete.

Maybe "when this action is completed"?

Documentation/admin-guide/cgroup-v2.rst All cgroup core files are prefixed with "cgroup." · line 909

Quoted 2 lines

Documentation/admin-guide/cgroup-v2.rst +2 -0

+	is complete, the "frozen" value in the cgroup.events control file
+	will be updated and the corresponding notification will be issued.

Can you please clarify how exactly cgroup.events would be updated?

Documentation/admin-guide/cgroup-v2.rst All cgroup core files are prefixed with "cgroup." · line 911

Quoted 1 line

Documentation/admin-guide/cgroup-v2.rst +1 -0

+	Cgroup can be frozen either by its own settings, either by settings

^ A cgroup ... and maybe there are more "a" and "the" that should be
fixed, it's hard for me to tell.

Also, I believe "either ..., or ..." sounds better than "either ..., either ..."

Documentation/admin-guide/cgroup-v2.rst All cgroup core files are prefixed with "cgroup." · line 917

Quoted 6 lines

Documentation/admin-guide/cgroup-v2.rst +6 -0

+	of any ancestor cgroups. If any of ancestor cgroups is frozen, the
+	cgroup will remain frozen.
+
+	Processes in the frozen cgroup can be killed by a fatal signal.
+	They also can enter and leave a frozen cgroup: either by an explicit
+	move by a user, either if freezing of the cgroup races with fork().

ditto

Quoted 7 lines

Documentation/admin-guide/cgroup-v2.rst All cgroup core files are prefixed with "cgroup." · line 918

Quoted 1 line

Documentation/admin-guide/cgroup-v2.rst +1 -0

+	If a cgroup is moved to a frozen cgroup, it stops. If a process is

^ process?

Documentation/admin-guide/cgroup-v2.rst All cgroup core files are prefixed with "cgroup." · line 919

Quoted 1 line

Documentation/admin-guide/cgroup-v2.rst +1 -0

+	moving out of a frozen cgroup, it becomes running.

^ moved

Hello, Mike!

Thanks for the review! I agree with all comments above; fixes queued for v4.

Quoted 1 line

Quoted 6 lines

Documentation/admin-guide/cgroup-v2.rst All cgroup core files are prefixed with "cgroup." · line 923

Quoted 3 lines

Documentation/admin-guide/cgroup-v2.rst +3 -0

+	Frozen status of a cgroup doesn't affect any cgroup tree operations:
+	it's possible to delete a frozen (and empty) cgroup, as well as
+	create new sub-cgroups.

Maybe it's also worth adding that freezing a process has no effect on its memory consumption, at least directly.

Hm, isn't it the expected behavior?

In any case, I assume that cgroup.freeze knob description is not the best place for a such explanations. Maybe it's better to add a standalone paragraph with the description of the frozen process state, what's expected to work, what's not, etc. I'd return to this question a bit later, when we'll agree on the user interface and the implementation.

Thanks!