sched: Relax a restriction in sched_rt_can_attach()
PATCH 1/1 sched: Relax a restriction in sched_rt_can_attach()
It's allowed to promote a task from normal to realtime after it has been attached to a non-root cgroup, but it will fail if the attaching happens after it has become realtime. I don't see how this restriction is useful.
We are moving toward unified hierarchy where all the cgroup controllers are bound together, so it would make cgroups easier to use if we have less restrictions on attaching tasks between cgroups.
--- kernel/sched/core.c | 14 ++++++-------- 1 file changed, 6 insertions(+), 8 deletions(-)
kernel/sched/core.c +6 -8
@@ -7800,6 +7800,11 @@ static int sched_rt_global_constraints(void) return ret; } + +static int sched_rt_can_attach (struct task_group *tg,struct task_struct *tsk) +{ +return 1 ; +}#endif /* CONFIG_RT_GROUP_SCHED */ static int sched_dl_global_validate (void )
@@ -8002,16 +8007,9 @@ static int cpu_cgroup_can_attach(struct cgroup_subsys_state *css, {struct task_struct *task; -cgroup_taskset_for_each (task, tset) { -#ifdef CONFIG_RT_GROUP_SCHED +cgroup_taskset_for_each (task, tset)if (!sched_rt_can_attach (css_tg (css), task))return -EINVAL ; -#else -/* We don't support RT-tasks being in separate groups */ -if (task->sched_class != &fair_sched_class) -return -EINVAL ; -#endif - }return 0 ; }
Signature · 1 line
1.9.1
Quoted 3 lines · Zefan Li
It's allowed to promote a task from normal to realtime after it has been attached to a non-root cgroup, but it will fail if the attaching happens after it has become realtime. I don't see how this restriction is useful.
In the CONFIG_RT_GROUP_SCHED case, promotion will fail is there is no bandwidth allocated.
Quoted 3 lines
We are moving toward unified hierarchy where all the cgroup controllers are bound together, so it would make cgroups easier to use if we have less restrictions on attaching tasks between cgroups.
Forcing group scheduling overhead on users if they want cpuset or memory cgroup functionality would be far from wonderful. Am I interpreting the implications of this unification/binding properly?
(I hope not, surely the plan is not to utterly _destroy_ cgroup utility)
-Mike
Quoted 8 lines · Mike Galbraith
Quoted 3 lines · Zefan Li
It's allowed to promote a task from normal to realtime after it has been attached to a non-root cgroup, but it will fail if the attaching happens after it has become realtime. I don't see how this restriction is useful.
In the
CONFIG_RT_GROUP_SCHEDcase, promotion will fail is there is no bandwidth allocated.
Right. I forgot to mention this patch affects !CONFIG_RT_GROUP_SCHED only, though it should be obvious by reading the change.
Quoted 10 lines
Quoted 3 lines
We are moving toward unified hierarchy where all the cgroup controllers are bound together, so it would make cgroups easier to use if we have less restrictions on attaching tasks between cgroups.
Forcing group scheduling overhead on users if they want cpuset or memory cgroup functionality would be far from wonderful. Am I interpreting the implications of this unification/binding properly?
(I hope not, surely the plan is not to utterly _destroy_ cgroup utility)
Some degree of flexibility is provided so that you may disable some controllers in a subtree. For example:
root ---> child1
(cpuset,memory,cpu) (cpuset,memory)
\
\-> child2
(cpu)Quoted 11 lines · Zefan Li
Quoted 10 lines
Quoted 3 lines
We are moving toward unified hierarchy where all the cgroup controllers are bound together, so it would make cgroups easier to use if we have less restrictions on attaching tasks between cgroups.
Forcing group scheduling overhead on users if they want cpuset or memory cgroup functionality would be far from wonderful. Am I interpreting the implications of this unification/binding properly?
(I hope not, surely the plan is not to utterly _destroy_ cgroup utility)
Quoted 8 lines
Some degree of flexibility is provided so that you may disable some controllers in a subtree. For example:
root ---> child1
(cpuset,memory,cpu) (cpuset,memory)\ \-> child2 (cpu)
Whew, that's a relief. Thanks.
-Mike
Quoted 16 lines · Mike Galbraith
Quoted 14 lines · Zefan Li
Quoted 10 lines
Quoted 3 lines
We are moving toward unified hierarchy where all the cgroup controllers are bound together, so it would make cgroups easier to use if we have less restrictions on attaching tasks between cgroups.
Forcing group scheduling overhead on users if they want cpuset or memory cgroup functionality would be far from wonderful. Am I interpreting the implications of this unification/binding properly?
(I hope not, surely the plan is not to utterly _destroy_ cgroup utility)
Some degree of flexibility is provided so that you may disable some controllers in a subtree. For example:
Quoted 7 lines
Quoted 5 lines
root ---> child1
(cpuset,memory,cpu) (cpuset,memory)\ \-> child2 (cpu)Whew, that's a relief. Thanks.
But somehow I'm not feeling a whole lot better.
"May" means if you don't explicitly take some action to disable group scheduling, you get it (I don't care if I have an off button), but that would also seemingly mean that we would then have rt tasks in taskgroups with no bandwidth allocated, ie you have to make group scheduling for rt tasks meaningless until a bandwidth appeared, and to make bandwidth appear, you'd have to stop the world, distribute, continue, no?
The current "just say no" seems a lot more sensible.
-Mike
Quoted 13 lines
Quoted 10 lines
Quoted 8 lines
Some degree of flexibility is provided so that you may disable some controllers in a subtree. For example:
root ---> child1
(cpuset,memory,cpu) (cpuset,memory)\ \-> child2 (cpu)Whew, that's a relief. Thanks.
But somehow I'm not feeling a whole lot better.
Quoted 9 lines
"May" means if you don't explicitly take some action to disable group scheduling, you get it (I don't care if I have an off button), but that would also seemingly mean that we would then have rt tasks in taskgroups with no bandwidth allocated, ie you have to make group scheduling for rt tasks meaningless until a bandwidth appeared, and to make bandwidth appear, you'd have to stop the world, distribute, continue, no?
The current "just say no" seems a lot more sensible.
I just realized we allow removing/adding controllers from/to cgroups while there are tasks in them, which isn't safe unless we eliminate all can_attach callbacks. We've done so for some cgroup subsystems, but there are still a few of them...
Quoted 20 lines · Zefan Li
Quoted 20 lines
Quoted 10 lines
Quoted 8 lines
Some degree of flexibility is provided so that you may disable some controllers in a subtree. For example:
root ---> child1
(cpuset,memory,cpu) (cpuset,memory)\ \-> child2 (cpu)Whew, that's a relief. Thanks.
But somehow I'm not feeling a whole lot better.
"May" means if you don't explicitly take some action to disable group scheduling, you get it (I don't care if I have an off button), but that would also seemingly mean that we would then have rt tasks in taskgroups with no bandwidth allocated, ie you have to make group scheduling for rt tasks meaningless until a bandwidth appeared, and to make bandwidth appear, you'd have to stop the world, distribute, continue, no?
Quoted 7 lines
Quoted 2 lines
The current "just say no" seems a lot more sensible.
I just realized we allow removing/adding controllers from/to cgroups while there are tasks in them, which isn't safe unless we eliminate all
can_attachcallbacks. We've done so for some cgroup subsystems, but there are still a few of them...
I was pondering the future (or so I thought), but seems it turned into the past while I wasn't looking. Oh well, you found a bug anyway.
-Mike
Peter Zijlstra
maintainerQuoted 8 lines · Zefan Li
Some degree of flexibility is provided so that you may disable some controllers in a subtree. For example:
root ---> child1
(cpuset,memory,cpu) (cpuset,memory)\ \-> child2 (cpu)
Uhm, how does that work? Would a task their effective cgroup be the first parent that has a controller enabled?
In particular, in your example, if T were part of child1, would its cpu controller be root?
Quoted 4 lines
I just realized we allow removing/adding controllers from/to cgroups while there are tasks in them, which isn't safe unless we eliminate all
can_attachcallbacks. We've done so for some cgroup subsystems, but there are still a few of them...
You can't remove can_attach(), we must be able to disallow joining a cgroup.
If that results in you not being able to change the cgroup setup with tasks in, so be it -- that seems like a sane restriction anyhow.
Quoted 11 lines · Peter Zijlstra
Quoted 8 lines · Zefan Li
Some degree of flexibility is provided so that you may disable some controllers in a subtree. For example:
root ---> child1
(cpuset,memory,cpu) (cpuset,memory)\ \-> child2 (cpu)
Quoted 5 lines
Uhm, how does that work? Would a task their effective cgroup be the first parent that has a controller enabled?
In particular, in your example, if T were part of child1, would its cpu controller be root?
That's what I'd hope for. I wanted to try that cgroup.subtree_control gizmo to see for myself, but I don't have one, and probably won't get one until I introduce systemd to my axe (again, it's a slow learner).
-Mike
Quoted 12 lines · Mike Galbraith
Quoted 11 lines · Peter Zijlstra
Quoted 8 lines · Zefan Li
Some degree of flexibility is provided so that you may disable some controllers in a subtree. For example:
root ---> child1
(cpuset,memory,cpu) (cpuset,memory)\ \-> child2 (cpu)
Quoted 5 lines
Quoted 5 lines
Uhm, how does that work? Would a task their effective cgroup be the first parent that has a controller enabled?
In particular, in your example, if T were part of child1, would its cpu controller be root?
correct.
Quoted 5 lines
That's what I'd hope for. I wanted to try that cgroup.subtree_control gizmo to see for myself, but I don't have one, and probably won't get one until I introduce systemd to my axe (again, it's a slow learner).
I'm testing in an environment without systemd.
You need to mount cgroup with a special option:
# mount -t cgroup -o __DEVEL__sane_behavior xxx /where
If a cgroup controller has already been mounted without this option, you won't see it in the unified hierarchy, so firstly you need to delete all cgroups in it and umount it.
Quoted 22 lines · Zefan Li
Quoted 12 lines · Mike Galbraith
Quoted 11 lines · Peter Zijlstra
Quoted 8 lines · Zefan Li
Some degree of flexibility is provided so that you may disable some controllers in a subtree. For example:
root ---> child1
(cpuset,memory,cpu) (cpuset,memory)\ \-> child2 (cpu)Quoted 5 lines
Quoted 5 lines
Uhm, how does that work? Would a task their effective cgroup be the first parent that has a controller enabled?
In particular, in your example, if T were part of child1, would its cpu controller be root?
correct.
Quoted 1 line
Quoted 6 lines
Quoted 4 lines
That's what I'd hope for. I wanted to try that cgroup.subtree_control gizmo to see for myself, but I don't have one, and probably won't get one until I introduce systemd to my axe (again, it's a slow learner).
I'm testing in an environment without systemd.
Lucky you.
Quoted 7 lines
You need to mount cgroup with a special option:
# mount -t cgroup -o __DEVEL__sane_behavior xxx /whereIf a cgroup controller has already been mounted without this option, you won't see it in the unified hierarchy, so firstly you need to delete all cgroups in it and umount it.
Yeah, I found the flag, and systemd is indeed in the way. You already verified what subtree_control does, so I needn't squabble with the vile thing over cgroups possession... immediately anyway.
-Mike
Quoted 22 lines · Peter Zijlstra
Quoted 8 lines · Zefan Li
Some degree of flexibility is provided so that you may disable some controllers in a subtree. For example:
root ---> child1
(cpuset,memory,cpu) (cpuset,memory)\ \-> child2 (cpu)Uhm, how does that work? Would a task their effective cgroup be the first parent that has a controller enabled?
In particular, in your example, if T were part of child1, would its cpu controller be root?
Quoted 4 lines
I just realized we allow removing/adding controllers from/to cgroups while there are tasks in them, which isn't safe unless we eliminate all
can_attachcallbacks. We've done so for some cgroup subsystems, but there are still a few of them...
Quoted 6 lines
You can't remove
can_attach(), we must be able to disallow joining a cgroup.If that results in you not being able to change the cgroup setup with tasks in, so be it -- that seems like a sane restriction anyhow.
I wasn't thinking about removing can_attach() before I noticed this issue.
But I was wondering if we can change the default value of cpu.rt_runtime_us from 0 to -1? So by default the RT tasks can be attached to a newly-created cgroup without users having to make any configuration, and those tasks are confined by the parent cgroup, which is what we have with cfs bw control. This require some changes to the code, but I guess it's do-able?
Peter Zijlstra
maintainerQuoted 5 lines · Zefan Li
But I was wondering if we can change the default value of cpu.rt_runtime_us from 0 to -1? So by default the RT tasks can be attached to a newly-created cgroup without users having to make any configuration, and those tasks are confined by the parent cgroup, which is what we have with cfs bw control. This require some changes to the code, but I guess it's do-able?
Its tricky.
Imagine:
root
/ \
A B
/ \ / \
a1 a2 b1 b2Now if they all have -1, I cannot set a bw on any except the leaf nodes ([ab][12]). Because the sum of child bw must strictly be smaller or equal to the parent bandwidth, and -1 if effective inf.
Similarly, if A has bw enabled I cannot create a new child with -1. Because above.
Now you can kludge around some of this, for example you can make the default depend on the parent setting etc.. But that's horribly inconsistent.
So I really prefer not to go that way; if people use RR/FIFO they had better bloody know what they're doing; which includes setting up the system.
The whole RR/FIFO thing is so enormously broken (by definition; this truly is unfixable) that you simply _cannot_ automate it.
Tejun Heo
maintainerHello, Peter.
Quoted 12 lines · Peter Zijlstra
Imagine:
root / \ A B / \ / \ a1 a2 b1 b2Now if they all have -1, I cannot set a bw on any except the leaf nodes ([ab][12]). Because the sum of child bw must strictly be smaller or equal to the parent bandwidth, and -1 if effective inf.
Quoted 6 lines
Similarly, if A has bw enabled I cannot create a new child with -1. Because above.
Now you can kludge around some of this, for example you can make the default depend on the parent setting etc.. But that's horribly inconsistent.
I don't think we can kludge this. For all other resources, we're defining the limits that can't be crossed so nesting them w/ -1 by default is fine. RR slices are different it that we're really slicing up and guaranteeing a portion of something finite, so unlimited by default thing doesn't really work here.
Quoted 3 lines
So I really prefer not to go that way; if people use RR/FIFO they had better bloody know what they're doing; which includes setting up the system.
The problem is that this is tied to the normal cpu controller. Users who don't have any intention of mucking with RT scheduling end up being dragged into it. Given the strict nature of RR slicing, I'm don't even think it's actually useful to make the slicing hierarchical. From cgroup's POV, it'd be best if RR slicing can be detached.
Quoted 2 lines
The whole RR/FIFO thing is so enormously broken (by definition; this truly is unfixable) that you simply _cannot_ automate it.
Yeah, exactly.
Thanks.
Signature · 1 line
tejun
Peter Zijlstra
maintainerQuoted 9 lines · Tejun Heo
Quoted 3 lines
Now you can kludge around some of this, for example you can make the default depend on the parent setting etc.. But that's horribly inconsistent.
I don't think we can kludge this. For all other resources, we're defining the limits that can't be crossed so nesting them w/ -1 by default is fine. RR slices are different it that we're really slicing up and guaranteeing a portion of something finite, so unlimited by default thing doesn't really work here.
Note that you _could_ do the same thing with IO bandwidth; esp. with these modern no-seek-penalty devices this could make sense.
Quoted 10 lines
Quoted 3 lines
So I really prefer not to go that way; if people use RR/FIFO they had better bloody know what they're doing; which includes setting up the system.
The problem is that this is tied to the normal cpu controller. Users who don't have any intention of mucking with RT scheduling end up being dragged into it. Given the strict nature of RR slicing, I'm don't even think it's actually useful to make the slicing hierarchical. From cgroup's POV, it'd be best if RR slicing can be detached.
Like in the other mail; hierarchy still makes perfect sense for the container case.
Quoted 4 lines
Quoted 2 lines
The whole RR/FIFO thing is so enormously broken (by definition; this truly is unfixable) that you simply _cannot_ automate it.
Yeah, exactly.
I don't think you're quite agreeing to the same reasons I am. My main objection to the whole SCHED_RR/FIFO thing as defined by POSIX is that it does not in fact allow the OS to do what an OS _should_ do, namely resource arbitration and control.
The whole rt-cgroup controller tries to somewhat contain that, but fundamentally once you use RR/FIFO you've given up your system to userspace control -- which btw is why its usually limited to root.
SCHED_DEADLINE avoids all these problems, at the cost of a more complex setup.
But the fact that both need fixed portions of a limited total does not in fact mean they're broken.
Tejun Heo
maintainerHello, Peter.
Quoted 8 lines · Peter Zijlstra
Quoted 5 lines
I don't think we can kludge this. For all other resources, we're defining the limits that can't be crossed so nesting them w/ -1 by default is fine. RR slices are different it that we're really slicing up and guaranteeing a portion of something finite, so unlimited by default thing doesn't really work here.
Note that you _could_ do the same thing with IO bandwidth; esp. with these modern no-seek-penalty devices this could make sense.
Yeah, maybe. It currently is too unpredictable to do that (at least from OS side w/ all the layering) but that is a possibility.
Quoted 9 lines
Quoted 6 lines
The problem is that this is tied to the normal cpu controller. Users who don't have any intention of mucking with RT scheduling end up being dragged into it. Given the strict nature of RR slicing, I'm don't even think it's actually useful to make the slicing hierarchical. From cgroup's POV, it'd be best if RR slicing can be detached.
Like in the other mail; hierarchy still makes perfect sense for the container case.
We'd still need an on-demand arbitration mechanism across containers no matter what we do which might as well take care of everything. But please see below.
Quoted 14 lines
Quoted 4 lines
Quoted 2 lines
The whole RR/FIFO thing is so enormously broken (by definition; this truly is unfixable) that you simply _cannot_ automate it.
Yeah, exactly.
I don't think you're quite agreeing to the same reasons I am. My main objection to the whole SCHED_RR/FIFO thing as defined by POSIX is that it does not in fact allow the OS to do what an OS _should_ do, namely resource arbitration and control.
The whole rt-cgroup controller tries to somewhat contain that, but fundamentally once you use RR/FIFO you've given up your system to userspace control -- which btw is why its usually limited to root.
Quoted 5 lines
SCHED_DEADLINEavoids all these problems, at the cost of a more complex setup.But the fact that both need fixed portions of a limited total does not in fact mean they're broken.
But that does make them pretty different from others. What bothers me the most about RR slices right now is that it's tightly coupled with the rest of cpu controller while having a very different set of characteristics. Maybe this is something mandated by the underlying structure and we have to live with it but it definitely isn't an ideal situation.
What I don't want to happen is controllers failing migrations willy-nilly for random reasons leaving users baffled, which we've actually been doing unfortunately. Maybe we need to deal with this fixed resource arbitration as a separate class and allow them to fail migration w/ -EBUSY.
Thanks.
Signature · 1 line
tejun
Peter Zijlstra
maintainerQuoted 6 lines · Tejun Heo
What I don't want to happen is controllers failing migrations willy-nilly for random reasons leaving users baffled, which we've actually been doing unfortunately. Maybe we need to deal with this fixed resource arbitration as a separate class and allow them to fail migration w/
-EBUSY.
Ah, _that_ was the problem.
Which is something created by this co-mounting of controllers.
You could of course store the ss-id of the failing operation in task_struct and have a file reporting the name of the ss-id.
That way, there is a simple way to find out which controller failed the migrate.
Tejun Heo
maintainerHello, Peter.
Quoted 10 lines · Peter Zijlstra
Quoted 5 lines · Tejun Heo
What I don't want to happen is controllers failing migrations willy-nilly for random reasons leaving users baffled, which we've actually been doing unfortunately. Maybe we need to deal with this fixed resource arbitration as a separate class and allow them to fail migration w/
-EBUSY.Ah, _that_ was the problem.
Which is something created by this co-mounting of controllers.
Yeah, partly, but also that it's an extra failure mode which isn't necessary for most controllers.
Quoted 5 lines
You could of course store the ss-id of the failing operation in
task_structand have a file reporting the name of the ss-id.That way, there is a simple way to find out which controller failed the migrate.
Given that the resources which can fail are very limited, I don't think we need that right now as long as we limit and document the possible failure cases clearly. Hopefully, this won't devolve into collection of arbitrary failures.
Thanks.
Signature · 1 line
tejun
Peter Zijlstra
maintainerQuoted 10 lines · Tejun Heo
Hello, Peter.
Quoted 7 lines · Peter Zijlstra
Quoted 5 lines · Tejun Heo
What I don't want to happen is controllers failing migrations willy-nilly for random reasons leaving users baffled, which we've actually been doing unfortunately. Maybe we need to deal with this fixed resource arbitration as a separate class and allow them to fail migration w/
-EBUSY.
Quoted 6 lines
Quoted 3 lines
Ah, _that_ was the problem.
Which is something created by this co-mounting of controllers.
Yeah, partly, but also that it's an extra failure mode which isn't necessary for most controllers.
I can agree with reducing failure modes, but we should not do it at the cost of functionality.
Quoted 10 lines
Quoted 5 lines
You could of course store the ss-id of the failing operation in
task_structand have a file reporting the name of the ss-id.That way, there is a simple way to find out which controller failed the migrate.
Given that the resources which can fail are very limited, I don't think we need that right now as long as we limit and document the possible failure cases clearly. Hopefully, this won't devolve into collection of arbitrary failures.
Right, but something like that would be fairly trivial to implement and would give immediate resolution.
For example:
$ echo 123 > /cgroups/monkey/business/tasks -EBUSY $ cat /cgroups/monkey/business/errno cpu:-EBUSY
(in fact, for a trivial implementation it doesn't matter which cgroup/errno you cat)
Tejun Heo
maintainerHello, Peter.
Quoted 10 lines · Peter Zijlstra
Quoted 4 lines
I just realized we allow removing/adding controllers from/to cgroups while there are tasks in them, which isn't safe unless we eliminate all
can_attachcallbacks. We've done so for some cgroup subsystems, but there are still a few of them...You can't remove
can_attach(), we must be able to disallow joining a cgroup.If that results in you not being able to change the cgroup setup with tasks in, so be it -- that seems like a sane restriction anyhow.
This is really an interface policy issue. For all other controllers, it's almost trivial to let organizational operations (setting up hierarchies, moving processes around) overrule controller configurations. The main benefit of doing this is that this decouples organizational operations from resource control. Users can depend on the fact that allowed organizational operations won't fail due to specific controller configuration issues.
This also works well with controllers accepting target configurations regardless of the current state and enforcing rules to converge to the configured state instead. e.g. if you set max memory lower than the currently used, the config will be accepted and the controller will keep trying to make the current state converge to the target state. This is important as rejecting configuration can lead to chasing game between configuration attempts and run-away resource consumption.
Now, RR slices are the special case here because it's inherently different from every other resource cgroup is concerned with. It simply doesn't fit into the same model that other resources follow. There are several options we can try.
1. Decouple RR slices from cpu controller. This would be the best route to follow. RR slices need a hard allocator no matter what we do. There isn't much point in imposing hierarchical structure on top of it.
2. Implement special case behavior so that it can follow the same model. e.g. resetting RR scheduling config when the effective cpu cgroup changes or carrying the amount of slice being consumed with the process being moved. No matter how this is done, it's gonna be a clear compromise as we're forcing this into the model which doesn't quite fit it. That said, given how RR slices are a special case to begin with, I think this can be acceptable.
3. Take compromise in the other direction - add exceptions to organizational operations but clearly limit the failure modes. We prolly want to structure code in a way to enforce this.
4. If #1 can be done in time but not right now, simply disallow any RR/FIFO in !root cgroups on the unified hierarchy for now.
What do you think?
Thanks.
Signature · 1 line
tejun
Peter Zijlstra
maintainerQuoted 14 lines · Tejun Heo
Hello, Peter.
Quoted 10 lines · Peter Zijlstra
Quoted 4 lines
I just realized we allow removing/adding controllers from/to cgroups while there are tasks in them, which isn't safe unless we eliminate all
can_attachcallbacks. We've done so for some cgroup subsystems, but there are still a few of them...You can't remove
can_attach(), we must be able to disallow joining a cgroup.If that results in you not being able to change the cgroup setup with tasks in, so be it -- that seems like a sane restriction anyhow.
Quoted 7 lines
This is really an interface policy issue. For all other controllers, it's almost trivial to let organizational operations (setting up hierarchies, moving processes around) overrule controller configurations. The main benefit of doing this is that this decouples organizational operations from resource control. Users can depend on the fact that allowed organizational operations won't fail due to specific controller configuration issues.
But but but... that doesn't make any damn sense! Why would you want to do something mad like that?
To me the organization is very much part of the control structure. It cannot be an invariant. Treating it like that destroys the whole notion of a hierarchy.
Quoted 3 lines
This also works well with controllers accepting target configurations regardless of the current state and enforcing rules to converge to the configured state instead.
I think we had a long discussion on that which we never finished. I'm not much for converging to a state. Either it can or it can not and you hard fail.
With this soft lets just accept any old crap mentality you cannot provide guarantees.
Quoted 5 lines
e.g. if you set max memory lower than the
currently used, the config will be accepted and the controller will keep trying to make the current state converge to the target state. This is important as rejecting configuration can lead to chasing game between configuration attempts and run-away resource consumption.
This is an entirely different issue; albeit with its own pitfalls, what if you put the max too low and you run into a never ending reclaim loop? Attempting to attain the unattainable.
Quoted 2 lines
Now, RR slices are the special case here because it's inherently different from every other resource cgroup is concerned with.
I don't think so, any controller which wants to carve up a fixed resource in non proportional ways is going to run into this.
Its just that you don't want this, but that doesn't render it less useful.
Quoted 8 lines
It
simply doesn't fit into the same model that other resources follow. There are several options we can try.1. Decouple RR slices from cpu controller. This would be the best route to follow. RR slices need a hard allocator no matter what we do. There isn't much point in imposing hierarchical structure on top of it.
The same is true of SCHED_DEADLINE, we hard divide a fixed amount. We've not currently exposed it to cgroups, but we want to eventually.
As to not having a hierarchy; you're the one destroying it by saying the organization should be decoupled from the controller.
And, no a hierarchy still makes perfect sense, think of containers, they might not even see the parent.
Quoted 3 lines
3. Take compromise in the other direction - add exceptions to organizational operations but clearly limit the failure modes. We prolly want to structure code in a way to enforce this.
I'm for failure modes as you should well now by know ;-)
I really think you're moving in the wrong direction with the whole cgroup stuff if you just want to willy nilly allow everything.
Also, who's the one doing a PID controller which will hard fail fork? How are you going to do away with can_attach() there? Surely you need to dis-allow another task joining when its at its maximum number of allowed PIDs, the same condition you're going to fail fork().
So no; hard failure is good and desired. It allows guarantees, which is a good and desired feature of control.
Tejun Heo
maintainerHello, Peter.
On Tue, May 05, 2015 at 05:11:13PM +0200, Peter Zijlstra wrote:
...
Quoted 6 lines
But but but... that doesn't make any damn sense! Why would you want to do something mad like that?
To me the organization is very much part of the control structure. It cannot be an invariant. Treating it like that destroys the whole notion of a hierarchy.
You and I don't really agree on this. The disagreement is fine but what I don't get is why this is such a big deal. How would it break the whole notion of a hierarchy? A user isn't allowed to esacpe the subhierarchy it's allowed in no matter what. Whether organizational operations supercedes configurations or not doesn't matter as long as the user is confined under the right hierarchy.
Furthermore, in majority of use cases, organizational operations are used to set up the hierarchy when starting up a group and then left alone. For stateful controller like memcg process migrations are inherently expensive and intrusive, so the usage model isn't arbitrary. This is a corner case issue and doesn't really affect the whole model.
Quoted 9 lines
Quoted 5 lines
e.g. if you set max memory lower than the
currently used, the config will be accepted and the controller will keep trying to make the current state converge to the target state. This is important as rejecting configuration can lead to chasing game between configuration attempts and run-away resource consumption.This is an entirely different issue; albeit with its own pitfalls, what if you put the max too low and you run into a never ending reclaim loop? Attempting to attain the unattainable.
That's an oom condition and memcg handles it accordingly.
Quoted 8 lines
Quoted 2 lines
Now, RR slices are the special case here because it's inherently different from every other resource cgroup is concerned with.
I don't think so, any controller which wants to carve up a fixed resource in non proportional ways is going to run into this.
Its just that you don't want this, but that doesn't render it less useful.
Well, of the resources that we handle right now, it is a special case and a sucky one at that because it ties itself to regular cpu controller which doesn't need that behavior.
Quoted 9 lines
Quoted 8 lines
It
simply doesn't fit into the same model that other resources follow. There are several options we can try.1. Decouple RR slices from cpu controller. This would be the best route to follow. RR slices need a hard allocator no matter what we do. There isn't much point in imposing hierarchical structure on top of it.
Quoted 5 lines
The same is true of
SCHED_DEADLINE, we hard divide a fixed amount. We've not currently exposed it to cgroups, but we want to eventually.As to not having a hierarchy; you're the one destroying it by saying the organization should be decoupled from the controller.
I don't get this part. How does making organization supercede configuration destroy hierarchy?
Quoted 2 lines
And, no a hierarchy still makes perfect sense, think of containers, they might not even see the parent.
The mode of configuration is different tho. No matter what we do, if we want to automate this sort of distribution with resource as limited as realtime slices, it'll need a separate allocator which can carve out resources on demand. This can't be ratio-distributed or soft-capped and having to tie this together with regular cpu controller is annoying.
Quoted 8 lines
Quoted 3 lines
3. Take compromise in the other direction - add exceptions to organizational operations but clearly limit the failure modes. We prolly want to structure code in a way to enforce this.
I'm for failure modes as you should well now by know ;-)
I really think you're moving in the wrong direction with the whole cgroup stuff if you just want to willy nilly allow everything.
Well, let's agree to disagree on that one. It's not about allowing willy nilly everything but separating out the specification of intent from the current state and you also saw how coupling the two tightly messed up cpuset. It can make configuration tedious enough to the point where it becomes impractical to use under certain circumstances.
The thing is, allowing to specify configurations doesn't prevent the user from enforcing stricter rules. The current state is always visible to the user and if it fails to converge, the user can take whatever actions that it needs to take to remedy the situation.
Quoted 4 lines
Also, who's the one doing a PID controller which will hard fail fork? How are you going to do away with
can_attach()there? Surely you need to dis-allow another task joining when its at its maximum number of allowed PIDs, the same condition you're going to failfork().
It allows migrations into already capped cgroup. It just won't allow new forks. This isn't different from allowing limit to be lowered below the current and we *do* want that because otherwise it becomes a race between whoever is setting the config and whoever is consuming the resources. You always wanna be able to say "stop giving out resources now".
Quoted 2 lines
So no; hard failure is good and desired. It allows guarantees, which is a good and desired feature of control.
Isn't that too sweeping a statement? We want them in some places but not necessarily in all places. The hard failures aren't going away. They're just localized to specific areas where they're easier to handle.
Thanks.
Signature · 1 line
tejun
Peter Zijlstra
maintainerQuoted 11 lines · Tejun Heo
Hello, Peter.
On Tue, May 05, 2015 at 05:11:13PM +0200, Peter Zijlstra wrote:
...Quoted 6 lines
But but but... that doesn't make any damn sense! Why would you want to do something mad like that?
To me the organization is very much part of the control structure. It cannot be an invariant. Treating it like that destroys the whole notion of a hierarchy.
Quoted 6 lines
You and I don't really agree on this. The disagreement is fine but what I don't get is why this is such a big deal. How would it break the whole notion of a hierarchy? A user isn't allowed to esacpe the subhierarchy it's allowed in no matter what. Whether organizational operations supercedes configurations or not doesn't matter as long as the user is confined under the right hierarchy.
I really don't get what you're saying there. If its not allowed to 'escape' there must be some equivalent of can_attach().
Otherwise you simply cannot reject the move.
Quoted 6 lines
Furthermore, in majority of use cases, organizational operations are used to set up the hierarchy when starting up a group and then left alone. For stateful controller like memcg process migrations are inherently expensive and intrusive, so the usage model isn't arbitrary. This is a corner case issue and doesn't really affect the whole model.
Again, I don't follow, so why is can_attach() bad?
Quoted 9 lines
Quoted 5 lines
I don't think so, any controller which wants to carve up a fixed resource in non proportional ways is going to run into this.
Its just that you don't want this, but that doesn't render it less useful.
Well, of the resources that we handle right now, it is a special case and a sucky one at that because it ties itself to regular cpu controller which doesn't need that behavior.
It doesn't 'tie' itself to the cpu controller, its a fundamental part of the cpu controller. The cpu controller is about all computation time, RR/FIFO is a very much part of that.
And RR/FIFO is extra special in that if you grant a process that it can suck your machine dry of this time. This is why you must configure it.
People should not unknowingly let programs use RR/FIFO. Also what sorts of 'problems' are people having because of this? What kind of applications 'require' RR/FIFO on a normal desktop?
Quoted 5 lines
Quoted 2 lines
As to not having a hierarchy; you're the one destroying it by saying the organization should be decoupled from the controller.
I don't get this part. How does making organization supercede configuration destroy hierarchy?
If you want to unconditionally allow task migration between groups, the hierarchy doesn't actually mean anything.
You can't enforce hierarchical constraints. Which to me is the entire point of having a hierarchy.
Quoted 7 lines
Quoted 2 lines
And, no a hierarchy still makes perfect sense, think of containers, they might not even see the parent.
The mode of configuration is different tho. No matter what we do, if we want to automate this sort of distribution with resource as limited as realtime slices, it'll need a separate allocator which can carve out resources on demand.
But you don't want to automate, full stop.
Quoted 3 lines
This can't be ratio-distributed or
soft-capped and having to tie this together with regular cpu controller is annoying.
Welcome to actual world issues. Stop pretending this stuff is easy and can be hidden from the user.
IF people want to use RR/FIFO they had better damn well know what they're doing. There is not way around that. There's just too many things that can go wrong with it.
If they don't want to deal with this problems, then tell them to go away. Do _NOT_ pretend its easy and fudge it for them.
This on-demand carving thing you mention, that's a _MASSIVE_ fudge. Just don't even go there.
Quoted 8 lines
Quoted 2 lines
I really think you're moving in the wrong direction with the whole cgroup stuff if you just want to willy nilly allow everything.
Well, let's agree to disagree on that one. It's not about allowing willy nilly everything but separating out the specification of intent from the current state and you also saw how coupling the two tightly messed up cpuset. It can make configuration tedious enough to the point where it becomes impractical to use under certain circumstances.
Well, no I didn't see how cpusets was messed up. You see that is where we start to disagree.
The improvement I wanted to cpusets was to simply disallow hotplug when there were tasks that could not go elsewhere.
Quoted 4 lines
The thing is, allowing to specify configurations doesn't prevent the user from enforcing stricter rules. The current state is always visible to the user and if it fails to converge, the user can take whatever actions that it needs to take to remedy the situation.
Right, so how about failing hotplug if there's (user) tasks pinned to a cpu? That's clearly visible and the user can go fix it if he really wants to do the unplug.
That's a very similar thing, but you've argued against it.
That said, this is not the point we're now arguing about; I want the hierarchy to actually mean something, and the only way to do that is to allow can_attach().
Without can_attach() one cannot provide hierarchical constraints.
Quoted 6 lines
Quoted 4 lines
Also, who's the one doing a PID controller which will hard fail fork? How are you going to do away with
can_attach()there? Surely you need to dis-allow another task joining when its at its maximum number of allowed PIDs, the same condition you're going to failfork().It allows migrations into already capped cgroup.
OMFG, that's so broken. This basically renders the entire cap useless.
So you now have: no more than 'N' tasks, except <big-gaping-hole>.
Quoted 6 lines
It just won't allow
new forks. This isn't different from allowing limit to be lowered below the current and we *do* want that because otherwise it becomes a race between whoever is setting the config and whoever is consuming the resources. You always wanna be able to say "stop giving out resources now".
Ah, that is what you've been trying to say with your memcg example. Well see this cannot work for realtime (and anybody else who wants to provide actual guarantees).
You simply cannot lower the max below the current usage, end of story. Because it will _NOT_ converge. Tasks were promised that time and will continue using it.
If you want to lower it, first take some tasks out. Idem the cpu affinity vs hotplug.
Same for your PID controller btw, it will NOT converge, tasks won't magically go away just because you want them to.
Also, there is no problem failing any of these setting, its 'obvious' what the problem is. When they return -EBUSY or whatnot, the resource is taken and you need to go free some up.
Quoted 7 lines
Quoted 2 lines
So no; hard failure is good and desired. It allows guarantees, which is a good and desired feature of control.
Isn't that too sweeping a statement? We want them in some places but not necessarily in all places. The hard failures aren't going away. They're just localized to specific areas where they're easier to handle.
Easier how? I'm really not seeing how any of this is making things easier for anybody.
All I'm seeing is that you're making cgroups useless for people who want to guarantee things (eg. the realtime people).
Are you really going to force us to abandon cgroups and invent yet another grouping thing?
Quoted 11 lines · Peter Zijlstra
Quoted 9 lines · Tejun Heo
Quoted 3 lines · Peter Zijlstra
So no; hard failure is good and desired. It allows guarantees, which is a good and desired feature of control.
Isn't that too sweeping a statement? We want them in some places but not necessarily in all places. The hard failures aren't going away. They're just localized to specific areas where they're easier to handle.
Quoted 5 lines
Easier how? I'm really not seeing how any of this is making things easier for anybody.
All I'm seeing is that you're making cgroups useless for people who want to guarantee things (eg. the realtime people).
I fully agree and after reading through this thread I really have to say that this whole notion of relax the admission control and then try to magically converge to the resource limits is horrible in all aspects.
Hierarchies must have a strictly inherited and overall consistent resource management and therefor resource limitation. Otherwise they are just useless.
The idea of allowing overcommitment and magically converging to back to the limits yells heuristics all over the place and we all know how reliable heuristics are.
Tejun, you try to make the whole configuration and placement simpler for the user, but all you achieve is that you act like all these politicians who promise tax cuts and whatever and forget about them once the elections are over. How is that going to make stuff simpler for users/admins? Not at all.
Instead of failing hard at placement/configuration time they get surprised by hard to understand fallout of magic convergence heuristics. That's crap and no matter how you argue it stays crap.
As Peter said several times: hard failure is good and desired. It's a very clear information on which people can act on. If the failures modes are nilly-willy today, as you wrote somewhere, then we need to fix that and make them consistent and understandable and not replace them by half baken heuristics which postpone the failure to some point where it is even less understandable.
If there are issues with run-away problems, i.e. upping a resource limit which gets eaten up from the existing tasks before you can admit a new one, then your magic convergence thing is again the wrong answer. The right approach is:
1) Up the limit and make a reservation at the same time
2) Admit the new task and allow it to consume the reservation
3) Set it effective
You can apply this to ALL sorts of resource controllers and you give the user a very simple to understand mechanism to control and configure his system.
Quoted 2 lines
Are you really going to force us to abandon cgroups and invent yet another grouping thing?
Sigh no. I think cgroups can be fixed, if we just adhere to the basic principles of hierarchical resource management and remove/reject all magic "we'll fix that for you" nonsense.
Thanks,
tglx
Tejun Heo
maintainerHello, Thomas.
Quoted 4 lines · Thomas Gleixner
I fully agree and after reading through this thread I really have to say that this whole notion of relax the admission control and then try to magically converge to the resource limits is horrible in all aspects.
This comes down to controllers allowing limits to be configured current usage. We need to allow and define what happens in that situation and moving a process into a full cgroup inherently follows the same pattern albeit from the other direction.
Quoted 3 lines
The idea of allowing overcommitment and magically converging to back to the limits yells heuristics all over the place and we all know how reliable heuristics are.
It's not magic heuristics. This is a core part of normal operation.
Quoted 6 lines
As Peter said several times: hard failure is good and desired. It's a very clear information on which people can act on. If the failures modes are nilly-willy today, as you wrote somewhere, then we need to fix that and make them consistent and understandable and not replace them by half baken heuristics which postpone the failure to some point where it is even less understandable.
There are no such magic heuristics because controllers need well defined behaviors when current is above limit anyway and behave exactly the same way no matter how that state is reached. For resources like RR slices, this doesn't work and that's why this is an issue, so yeah this is the process of finding out what must be able to fail.
Quoted 8 lines
If there are issues with run-away problems, i.e. upping a resource limit which gets eaten up from the existing tasks before you can admit a new one, then your magic convergence thing is again the wrong answer. The right approach is:
1) Up the limit and make a reservation at the same time
2) Admit the new task and allow it to consume the reservation
3) Set it effective
I don't really think this is a scenario we need to worry about. If we choose to fail migration, let's just fail it. There's no point in building a mechanism to work around malbehavior from its users.
Quoted 6 lines
Quoted 2 lines
Are you really going to force us to abandon cgroups and invent yet another grouping thing?
Sigh no. I think cgroups can be fixed, if we just adhere to the basic principles of hierarchical resource management and remove/reject all magic "we'll fix that for you" nonsense.
So, let's do -EBUSY for hard resource failures which have to be exact.
Thanks.
Signature · 1 line
tejun
Quoted 11 lines · Tejun Heo
Quoted 6 lines · Thomas Gleixner
As Peter said several times: hard failure is good and desired. It's a very clear information on which people can act on. If the failures modes are nilly-willy today, as you wrote somewhere, then we need to fix that and make them consistent and understandable and not replace them by half baken heuristics which postpone the failure to some point where it is even less understandable.
There are no such magic heuristics because controllers need well defined behaviors when current is above limit anyway and behave exactly the same way no matter how that state is reached. For
How would something go above limit in the first place if your resource management is done proper?
If a group has a resource limit, then it is not allowed to exceed that resource. So any attempt to use more resources must fail, period. There is no way to go above the limit.
If you try to lower the limits of an existing group below the level which is already used, then this limit restriction attempt must fail.
That's the basic principle of resource management. And if you try to avoid them, then you have a massive design failure. It's that simple.
Thanks,
tglx
Tejun Heo
maintainerHello, again.
Quoted 4 lines · Peter Zijlstra
I really don't get what you're saying there. If its not allowed to 'escape' there must be some equivalent of
can_attach().Otherwise you simply cannot reject the move.
A given user isn't allowed to move processes into a cgroup outside its subhierarchy and the hierarchical resource control keeps the subhierarchy under the limits no matter what the user does inside it. Whether can_attach can fail or not is peripheral in this sense - if a user can move processes into a cgroup outside its allowed scope, the user can already escape regardless of the specifics of configuration.
Any user of cgroups should be confined to its scope and when it's confined that way, the hierarchical limits are enforced no matter what happens in its subhierarchy.
Quoted 8 lines
Quoted 6 lines
Furthermore, in majority of use cases, organizational operations are used to set up the hierarchy when starting up a group and then left alone. For stateful controller like memcg process migrations are inherently expensive and intrusive, so the usage model isn't arbitrary. This is a corner case issue and doesn't really affect the whole model.
Again, I don't follow, so why is
can_attach()bad?
It's more like can_attach failures don't add much for other controllers. Please see below.
Quoted 3 lines
People should not unknowingly let programs use RR/FIFO. Also what sorts of 'problems' are people having because of this? What kind of applications 'require' RR/FIFO on a normal desktop?
The cases I hear about are mostly audio applications which end up in whatever default cgroups other applications are put in w/o an easy way to configure the hierarchy for RR slices. As I wrote way back, if these can't be decoupled, whoever is setting up cpu cgroup hierarchies will also have to take part in distributing realtime slices.
This might not necessarily be a bad thing. It's just different from everything else cgroups deal with at this point.
Quoted 8 lines
Quoted 2 lines
I don't get this part. How does making organization supercede configuration destroy hierarchy?
If you want to unconditionally allow task migration between groups, the hierarchy doesn't actually mean anything.
You can't enforce hierarchical constraints. Which to me is the entire point of having a hierarchy.
No, hierarchy still puts restrictions on who can do what where. Whether organization operations supercede configurations or not doens't affect this at all. Again, if you can stow away processes out of your domain, you're escaping the hierarchical constrasints all the same. Delegations need to scoped no matter what.
Quoted 11 lines
Quoted 3 lines
This can't be ratio-distributed or
soft-capped and having to tie this together with regular cpu controller is annoying.Welcome to actual world issues. Stop pretending this stuff is easy and can be hidden from the user.
IF people want to use RR/FIFO they had better damn well know what they're doing. There is not way around that. There's just too many things that can go wrong with it.
Quoted 5 lines
If they don't want to deal with this problems, then tell them to go away. Do _NOT_ pretend its easy and fudge it for them.
This on-demand carving thing you mention, that's a _MASSIVE_ fudge. Just don't even go there.
How is on-demand allocation fudging? You can do it manually or you can have policies set up to allocate the specific resource. This is really beside the point tho. What I was trying to say was that this takes a different approach from other non-hard resources.
Quoted 8 lines
Quoted 5 lines
Well, let's agree to disagree on that one. It's not about allowing willy nilly everything but separating out the specification of intent from the current state and you also saw how coupling the two tightly messed up cpuset. It can make configuration tedious enough to the point where it becomes impractical to use under certain circumstances.
Well, no I didn't see how cpusets was messed up. You see that is where we start to disagree.
Yeah, seems that way. Let's agree to disagree here.
Quoted 2 lines
The improvement I wanted to cpusets was to simply disallow hotplug when there were tasks that could not go elsewhere.
Would that mean we're also gonna disallow hotunplug if some threads are pinned to that cpu? And the kernel would still be changing configurations in an non-reversible way. Again, how does that jive with plain affinities?
Quoted 5 lines
That said, this is not the point we're now arguing about; I want the hierarchy to actually mean something, and the only way to do that is to allow
can_attach().Without
can_attach()one cannot provide hierarchical constraints.
I don't think this is the point either. The point is how to deal with hard resources that can't be permissive by default.
Quoted 10 lines
Quoted 6 lines
Quoted 4 lines
Also, who's the one doing a PID controller which will hard fail fork? How are you going to do away with
can_attach()there? Surely you need to dis-allow another task joining when its at its maximum number of allowed PIDs, the same condition you're going to failfork().It allows migrations into already capped cgroup.
OMFG, that's so broken. This basically renders the entire cap useless.
So you now have: no more than 'N' tasks, except <big-gaping-hole>.
We need that "hole" anyway as I wrote below. The rule is "no new fork if there are more than N tasks in the group" and that's it.
...
Quoted 7 lines
Ah, that is what you've been trying to say with your memcg example. Well see this cannot work for realtime (and anybody else who wants to provide actual guarantees).
You simply cannot lower the max below the current usage, end of story. Because it will _NOT_ converge. Tasks were promised that time and will continue using it.
So, this is the key issue. These resources are fundamentally different.
Quoted 5 lines
If you want to lower it, first take some tasks out. Idem the cpu affinity vs hotplug.
Same for your PID controller btw, it will NOT converge, tasks won't magically go away just because you want them to.
It won't automatically converge of course. It just won't allow new forks. Moving processes into the cgroup is at the same level as lowering the limit below current. If one is allowed, the other is allowed too and neither can allow the user to escape its hierarchical limit as long as the user is properly contained in its subhierarchy.
Quoted 3 lines
Also, there is no problem failing any of these setting, its 'obvious' what the problem is. When they return
-EBUSYor whatnot, the resource is taken and you need to go free some up.
Hmm... so, I kinda agree here. If we clearly define and constrain how we use -EBUSY (hard resources only), it can work out.
Thanks.
Signature · 1 line
tejun
Tejun Heo
maintainerHello, Mike.
Quoted 9 lines · Mike Galbraith
Quoted 9 lines
Quoted 8 lines
Some degree of flexibility is provided so that you may disable some controllers in a subtree. For example:
root ---> child1
(cpuset,memory,cpu) (cpuset,memory)\ \-> child2 (cpu)
Quoted 6 lines
Quoted 1 line
Whew, that's a relief. Thanks.
But somehow I'm not feeling a whole lot better.
"May" means if you don't explicitly take some action to disable group scheduling, you get it (I don't care if I have an off button), but that
In the new interface, hierarchy setup and controller configuration are two separate steps. Creating subhierarchy doesn't enable controller automatically and as long as specific controllers are concerned nothing changes when subhierarchy is created and processes are moved inbetween them. If control over specific resources is necessary in a given hierarchy, the matching controllers should be enabled explicitly.
Thanks.
Signature · 1 line
tejun