Writing "+pids" to cgroup.subtree_control flie yields EINVAL
Hello Tejun,
I was trying to do some simple testing ot the CPU controller that is merged into 4.15, and ran immediately into some confusion. In the root cgroup on a freshly booted 4.150-rc1, I try the following:
# pwd /sys/fs/cgroup/unified # echo '+cpu' > cgroup.subtree_control sh: echo: write error: Invalid argument
What am I missing> I presume I'm missing something obvious, although nothing jumped out at me as I read the cgroups-v2.txt file.
Cheers,
Michael
Signature · 3 lines
Michael Kerrisk Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/ Linux/UNIX System Programming Training: http://man7.org/training/
Hello, Michael.
Quoted 11 lines · Michael Kerrisk (man-pages)
I was trying to do some simple testing ot the CPU controller that is merged into 4.15, and ran immediately into some confusion. In the root cgroup on a freshly booted 4.150-rc1, I try the following:
# pwd /sys/fs/cgroup/unified # echo '+cpu' > cgroup.subtree_control sh: echo: write error: Invalid argumentWhat am I missing> I presume I'm missing something obvious, although nothing jumped out at me as I read the cgroups-v2.txt file.
Checking whether I messed up something really basic... hmmm doesn't seem that way. What do /sys/fs/cgroup/unified/cgroup.controllers and /proc/cgroups say?
Thanks.
Signature · 1 line
tejun
[dropping Lennart into CC]
Hello Tejun,
Quoted 12 lines · Tejun Heo
Hello, Michael.
Quoted 9 lines · Michael Kerrisk (man-pages)
I was trying to do some simple testing ot the CPU controller that is merged into 4.15, and ran immediately into some confusion. In the root cgroup on a freshly booted 4.150-rc1, I try the following:
# pwd /sys/fs/cgroup/unified # echo '+cpu' > cgroup.subtree_control sh: echo: write error: Invalid argument
Quoted 6 lines
Quoted 2 lines
What am I missing> I presume I'm missing something obvious, although nothing jumped out at me as I read the cgroups-v2.txt file.
Checking whether I messed up something really basic... hmmm doesn't seem that way. What do
/sys/fs/cgroup/unified/cgroup.controllersand/proc/cgroupssay?
Oh -- they're all sensible:
In the root cgroup:
# cat cgroup.controllers cpu io memory pids
$ cat /proc/cgroups #subsys_name hierarchy num_cgroups enabled cpuset 0 142 1 cpu 0 142 1 cpuacct 0 142 1 blkio 0 142 1 memory 0 142 1 devices 0 142 1 freezer 0 142 1 net_cls 0 142 1 perf_event 0 142 1 net_prio 0 142 1 hugetlb 0 142 1 pids 0 142 1
But, I through some trial and error and printk() I worked out
a) If I first move all tasks to the root cgroup, then I can write '+cpu' to the cgroup.subtree_control file in the root cgroup.
b) The reason for my initial problems was this test in the kernel in cpu_cgroup_can_attach():
#ifdef CONFIG_RT_GROUP_SCHED
if (!sched_rt_can_attach(css_tg(css), task))
return -EINVAL;
#else
/* We don't support RT-tasks being in separate groups */
if (task->sched_class != &fair_sched_class)
return -EINVAL;#endif
I don't have CONFIG_RT_GROUP_SCHED, and the second 'if' was yielding false because of some SCHED_RR processes that are in some of the nonroot cgroups created by systemd, namely:
# ps ax -L -o 'pid tid cls rtprio comm'|grep RR 685 723 RR 99 rtkit-daemon 972 979 RR 5 alsa-sink-ALC26 972 982 RR 5 alsa-source-ALC 1594 1597 RR 5 alsa-sink-ALC26 1594 1600 RR 5 alsa-source-ALC
So, one solution is to move those processes to the root cgroup, and then it's possible to write '+pids' to cgroup.subtree_control.
Is enabling CONFIG_RT_GROUP_SCHED also a solution? (I have not had a chance to test that yet.)
Anyway, it seems like this should be documented somewhere in the kernel Documentation files, since it may be that others will run into this as well. I'm not quite sure what should be added to the documentation. Do you have some idea?
Thanks,
Michael
Signature · 3 lines
Michael Kerrisk Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/ Linux/UNIX System Programming Training: http://man7.org/training/
Hello, Michael.
Quoted 10 lines · Michael Kerrisk (man-pages)
b) The reason for my initial problems was this test in the kernel in
cpu_cgroup_can_attach():#ifdef CONFIG_RT_GROUP_SCHED if (!sched_rt_can_attach(css_tg(css), task)) return -EINVAL; #else /* We don't support RT-tasks being in separate groups */ if (task->sched_class != &fair_sched_class) return -EINVAL;
Quoted 5 lines
#endif
I don't have
CONFIG_RT_GROUP_SCHED, and the second 'if' was yielding false because of someSCHED_RRprocesses that are in some of the nonroot cgroups created by systemd, namely:
I see. Thanks for tracking it down. Yeah, the RT side of things isn't too well thought out yet.
Quoted 9 lines
# ps ax -L -o 'pid tid cls rtprio comm'|grep RR 685 723 RR 99 rtkit-daemon 972 979 RR 5 alsa-sink-ALC26 972 982 RR 5 alsa-source-ALC 1594 1597 RR 5 alsa-sink-ALC26 1594 1600 RR 5 alsa-source-ALCSo, one solution is to move those processes to the root cgroup, and then it's possible to write '+pids' to cgroup.subtree_control.
You mean '+cpu", right?
Quoted 2 lines
Is enabling
CONFIG_RT_GROUP_SCHEDalso a solution? (I have not had a chance to test that yet.)
We aren't yet sure about how we should handle RT and haven't enabled RT part on cgroup2 yet. You can test the same scenario in cgroup1 tho and would have to configure RT shares all along the hierarchy to the leaf cgroup.
Quoted 4 lines
Anyway, it seems like this should be documented somewhere in the kernel Documentation files, since it may be that others will run into this as well. I'm not quite sure what should be added to the documentation. Do you have some idea?
I think the only thing we can say right now is that RT processes should be in the root cgroup. :(
Thanks a lot.
Signature · 1 line
tejun
Hello Tejun,
Quoted 29 lines · Tejun Heo
Hello, Michael.
Quoted 10 lines · Michael Kerrisk (man-pages)
b) The reason for my initial problems was this test in the kernel in
cpu_cgroup_can_attach():#ifdef CONFIG_RT_GROUP_SCHED if (!sched_rt_can_attach(css_tg(css), task)) return -EINVAL; #else /* We don't support RT-tasks being in separate groups */ if (task->sched_class != &fair_sched_class) return -EINVAL;Quoted 5 lines
#endif
I don't have
CONFIG_RT_GROUP_SCHED, and the second 'if' was yielding false because of someSCHED_RRprocesses that are in some of the nonroot cgroups created by systemd, namely:I see. Thanks for tracking it down. Yeah, the RT side of things isn't too well thought out yet.
Quoted 7 lines
# ps ax -L -o 'pid tid cls rtprio comm'|grep RR 685 723 RR 99 rtkit-daemon 972 979 RR 5 alsa-sink-ALC26 972 982 RR 5 alsa-source-ALC 1594 1597 RR 5 alsa-sink-ALC26 1594 1600 RR 5 alsa-source-ALC
Quoted 4 lines
Quoted 2 lines
So, one solution is to move those processes to the root cgroup, and then it's possible to write '+pids' to cgroup.subtree_control.
You mean '+cpu", right?
D'oh! Yes. I was also doing some tests with the 'pids' controller, and mistyped. (I see I even manage to mistitle this mail thread :-(.)
Quoted 7 lines
Quoted 2 lines
Is enabling
CONFIG_RT_GROUP_SCHEDalso a solution? (I have not had a chance to test that yet.)We aren't yet sure about how we should handle RT and haven't enabled RT part on cgroup2 yet. You can test the same scenario in cgroup1 tho and would have to configure RT shares all along the hierarchy to the leaf cgroup.
Okay. Thanks for the confirmation that this is at least surprising behavior.
Quoted 7 lines
Quoted 4 lines
Anyway, it seems like this should be documented somewhere in the kernel Documentation files, since it may be that others will run into this as well. I'm not quite sure what should be added to the documentation. Do you have some idea?
I think the only thing we can say right now is that RT processes should be in the root cgroup. :(
Okay. (Will you add that to the v2 doc file?)
Cheers,
Michael
--
Michael Kerrisk
Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/
Linux/UNIX System Programming Training: http://man7.org/training/
PATCH 1/1 cgroup: add warning about RT not being supported on cgroup2
From: Tejun Heo <[email protected]>
Date: Tue, 5 Dec 2017 09:10:17 -0800
We haven't yet figured out what to do with RT threads on cgroup2. Document the limitation.
--- Documentation/cgroup-v2.txt | 4 ++++ 1 file changed, 4 insertions(+)
Documentation/cgroup-v2.txt +4 -0
@@ -898,6 +898,10 @@ controller implements weight and absolute bandwidth limit models for normal scheduling policy and absolute bandwidth allocation model for realtime scheduling policy. +WARNING: cgroup2 doesn't yet support control of realtime processes and +the cpu controller can only be enabled when all RT processes are in +the root cgroup. + CPU Interface Files ~~~~~~~~~~~~~~~~~~~
Signature · 1 line
2.9.5
Re: [PATCH cgroup/for-4.15-fixes] cgroup: add warning about RT not being supported on cgroup2
Hello Tejun,
Quoted 17 lines · Tejun Heo
From: Tejun Heo <[email protected]>
Date: Tue, 5 Dec 2017 09:10:17 -0800We haven't yet figured out what to do with RT threads on cgroup2. Document the limitation.
Signed-off-by Tejun Heo <[email protected]>--- Documentation/cgroup-v2.txt | 4 ++++ 1 file changed, 4 insertions(+)Documentation/cgroup-v2.txt +0
-0
Documentation/cgroup-v2.txt controller implements weight and absolute bandwidth limit models for · line 904
Quoted 7 lines
Documentation/cgroup-v2.txt +4
-0@@ -898,6 +898,10 @@ controller implements weight and absolute bandwidth limit models for normal scheduling policy and absolute bandwidth allocation model for realtime scheduling policy. +WARNING: cgroup2 doesn't yet support control of realtime processes and +the cpu controller can only be enabled when all RT processes are in +the root cgroup. +
The documentation file already carries this useful text:
[[
During transition to v2, system management software might still automount the v1 cgroup filesystem and so hijack all controllers during boot, before manual intervention is possible. To make testing and experimenting easier, the kernel parameter cgroup_no_v1= allows disabling controllers in v1 and make them always available in v2. ]]
In that spirit, it may be worth alerting the reader that "system management software" might already have put some RT processes in nonroot cgroups. Maybe something like this:
[[
WARNING: cgroup2 doesn't yet support control of realtime processes and the cpu controller can only be enabled when all RT processes are in the root cgroup. Be aware that that system management software may already have placed RT processes into nonroot cgroups during the system boot process, and these processes may need to be moved to the root cgroup before the cpu controller can be enabled.
]]
Thanks,
Michael
Signature · 3 lines
Michael Kerrisk Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/ Linux/UNIX System Programming Training: http://man7.org/training/
PATCH v2 1/1 cgroup: add warning about RT not being supported on cgroup2
Quoted 1 line
From
f0a821260f24ea9378ed0e0c149e3bbc1a1bd008Mon Sep 17 00:00:00 2001
From: Tejun Heo <[email protected]>
Date: Tue, 5 Dec 2017 09:10:17 -0800
We haven't yet figured out what to do with RT threads on cgroup2. Document the limitation.
v2: Included the warning about system management software behavior as
suggested by Michael.
--- Documentation/cgroup-v2.txt | 7 +++++++ 1 file changed, 7 insertions(+)
Documentation/cgroup-v2.txt +7 -0
@@ -898,6 +898,13 @@ controller implements weight and absolute bandwidth limit models for normal scheduling policy and absolute bandwidth allocation model for realtime scheduling policy. +WARNING: cgroup2 doesn't yet support control of realtime processes and +the cpu controller can only be enabled when all RT processes are in +the root cgroup. Be aware that that system management software may +already have placed RT processes into nonroot cgroups during the +system boot process, and these processes may need to be moved to the +root cgroup before the cpu controller can be enabled. + CPU Interface Files ~~~~~~~~~~~~~~~~~~~
Signature · 1 line
2.9.5
Re: [PATCH v2 cgroup/for-4.15-fixes] cgroup: add warning about RT not being supported on cgroup2
Quoted 21 lines · Tejun Heo
From
f0a821260f24ea9378ed0e0c149e3bbc1a1bd008Mon Sep 17 00:00:00 2001
From: Tejun Heo <[email protected]>
Date: Tue, 5 Dec 2017 09:10:17 -0800We haven't yet figured out what to do with RT threads on cgroup2. Document the limitation.
v2: Included the warning about system management software behavior as
suggested by Michael.Signed-off-by Tejun Heo <[email protected]>--- Documentation/cgroup-v2.txt | 7 +++++++ 1 file changed, 7 insertions(+)Documentation/cgroup-v2.txt +0
-0
Documentation/cgroup-v2.txt controller implements weight and absolute bandwidth limit models for · line 903
Quoted 6 lines
Documentation/cgroup-v2.txt +3
-0@@ -898,6 +898,13 @@ controller implements weight and absolute bandwidth limit models for normal scheduling policy and absolute bandwidth allocation model for realtime scheduling policy. +WARNING: cgroup2 doesn't yet support control of realtime processes and +the cpu controller can only be enabled when all RT processes are in +the root cgroup. Be aware that that system management software may
Sorry, there was a fault in my text: s/that that/that/
Thanks,
Michael
Signature · 3 lines
Michael Kerrisk Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/ Linux/UNIX System Programming Training: http://man7.org/training/
Re: [PATCH v2 cgroup/for-4.15-fixes] cgroup: add warning about RT not being supported on cgroup2
Quoted 5 lines · Michael Kerrisk (man-pages)
Documentation/cgroup-v2.txt controller implements weight and absolute bandwidth limit models for · line 903
Quoted 3 lines
Documentation/cgroup-v2.txt +3
-0+WARNING: cgroup2 doesn't yet support control of realtime processes and +the cpu controller can only be enabled when all RT processes are in +the root cgroup. Be aware that that system management software maySorry, there was a fault in my text: s/that that/that/
Heh, fixed.
Thanks.
Signature · 1 line
tejun