autogroup: sched_setscheduler() fails

Hi,

With autogroup ON, sched_setscheduler() fails when I try to change the scheduling policy of a normal task to either RR or FIFO.

sched_setscheduler() returns -EPERM when it finds that the group doesn't have any rt bandwidth. This is expected because with autogroup, the task is in an autogroup for which task_group(p)->rt_bandwidth.rt_runtime is 0.

I guess the task needs to be moved to appropriate cgroup (from its current autogroup) when such policy change is requested, but I wasn't sure if group change from sched_setscheduler() can be achieved easily.

Right, autogroup should be excluding CONFIG_RT_GROUP_SCHED, since it doesn't currently support it.

sched: SCHED_AUTOGROUP selection excludes RT_GROUP_SCHED

Autogroup does not support realtime task groups, so make selection of SCHED_AUTOGROUP exclude RT_GROUP_SCHED.

Signed-off-by Mike Galbraith <[email protected]>

Reported-by Bharata B Rao <[email protected]>

---
 init/Kconfig |    5 +++++
 1 file changed, 5 insertions(+)

Index: linux-2.6/init/Kconfig
===================================================================

init/Kconfig +1 -0

@@ -699,6 +699,7 @@ config RT_GROUP_SCHED
        bool "Group scheduling for SCHED_RR/FIFO"
        depends on EXPERIMENTAL
        depends on CGROUP_SCHED
+       depends on !SCHED_AUTOGROUP
        default n
        help
          This feature lets you explicitly allocate real CPU bandwidth

Quoted 4 lines · Mike Galbraith

Autogroup does not support realtime task groups, so make selection of SCHED_AUTOGROUP exclude RT_GROUP_SCHED.

Uhm, no. The right way is to put tasks back into the root group and then perform sched_setscheduler().

Index: linux-2.6/kernel/sched_autogroup.c
===================================================================

kernel/sched_autogroup.c +9 -0

@@ -73,6 +73,15 @@ static inline struct autogroup *autogrou
 	ag->id = atomic_inc_return(&autogroup_seq_nr);
 	ag->tg = tg;
 	tg->autogroup = ag;
+#ifdef CONFIG_RT_GROUP_SCHED
+	/*
+	 * HACK: autogroup RT tasks run in the root task group.
+	 * This fools __sched_setscheduler() into proceeding on
+	 * so we can move the task to the appropriate runqueue
+	 * upon scheduling policy change.
+	 */
+	tg->rt_bandwidth.rt_runtime = RUNTIME_INF;
+#endif
 
 	return ag;
 

@@ -143,6 +152,15 @@ autogroup_move_group(struct task_struct
 	autogroup_kref_put(prev);
 }

Quoted 23 lines · Mike Galbraith

Index: linux-2.6/kernel/sched_autogroup.c
===================================================================

kernel/sched_autogroup.c +9 -0

@@ -73,6 +73,15 @@ static inline struct autogroup *autogrou
 	ag->id = atomic_inc_return(&autogroup_seq_nr);
 	ag->tg = tg;
 	tg->autogroup = ag;
+#ifdef CONFIG_RT_GROUP_SCHED
+	/*
+	 * HACK: autogroup RT tasks run in the root task group.
+	 * This fools __sched_setscheduler() into proceeding on
+	 * so we can move the task to the appropriate runqueue
+	 * upon scheduling policy change.
+	 */
+	tg->rt_bandwidth.rt_runtime = RUNTIME_INF;
+#endif
 
 	return ag;
 

@@ -143,6 +152,15 @@ autogroup_move_group(struct task_struct
 	autogroup_kref_put(prev);
 }

kernel/sched_autogroup.c autogroup_move_group(struct task_struct · line 163

Quoted 10 lines

kernel/sched_autogroup.c +9 -0

 
+static inline void
+autogroup_setscheduler(struct task_struct *p, int on_rq)
+{
+	if (p->sched_class->task_move_group)
+		p->sched_class->task_move_group(p, on_rq);
+	else
+		set_task_rq(p, task_cpu(p));
+}
+

IIUC, with the above changes you are actually queing the task into rt_rq of an autogroup. But the task's autogroup interface (/proc/<pid>/autogroup) allows you to control the bandwidth of only cfs_rq tasks, while the rt tasks in the group get RUNTIME_INF bandwidth.

I think what we need is a real group change here (which is difficult I think) and not just sched_class change.

Quoted 40 lines · Bharata B Rao

Quoted 23 lines · Mike Galbraith

Index: linux-2.6/kernel/sched_autogroup.c
===================================================================

kernel/sched_autogroup.c +9 -0

@@ -73,6 +73,15 @@ static inline struct autogroup *autogrou
 	ag->id = atomic_inc_return(&autogroup_seq_nr);
 	ag->tg = tg;
 	tg->autogroup = ag;
+#ifdef CONFIG_RT_GROUP_SCHED
+	/*
+	 * HACK: autogroup RT tasks run in the root task group.
+	 * This fools __sched_setscheduler() into proceeding on
+	 * so we can move the task to the appropriate runqueue
+	 * upon scheduling policy change.
+	 */
+	tg->rt_bandwidth.rt_runtime = RUNTIME_INF;
+#endif
 
 	return ag;
 

@@ -143,6 +152,15 @@ autogroup_move_group(struct task_struct
 	autogroup_kref_put(prev);
 }

kernel/sched_autogroup.c autogroup_move_group(struct task_struct · line 163

Quoted 10 lines

kernel/sched_autogroup.c +9 -0

 
+static inline void
+autogroup_setscheduler(struct task_struct *p, int on_rq)
+{
+	if (p->sched_class->task_move_group)
+		p->sched_class->task_move_group(p, on_rq);
+	else
+		set_task_rq(p, task_cpu(p));
+}
+

IIUC, with the above changes you are actually queing the task into rt_rq of an autogroup. But the task's autogroup interface (/proc/<pid>/autogroup) allows you to control the bandwidth of only cfs_rq tasks, while the rt tasks in the group get RUNTIME_INF bandwidth.

Quoted 2 lines

I think what we need is a real group change here (which is difficult I think) and not just sched_class change.

Unless I fscked up, set_task_rq() is the group change. As soon as the task's class changes, it'll be moved to the root_task_group.

Quoted 2 lines

Unless I fscked up, set_task_rq() is the group change.  As soon as the task's class changes, it'll be moved to the root_task_group.

Ok, this is what I understand, may be Peter Z can confirm...

set_task_rq() just changes the task's cfs_rq and rt_rq as per its task_group(). The normal way to change a task's group is to first change its cgroup pointer (task->cgroups, see cgroup_attach_task()) After this you change the runqueues by calling set_task_rq() which now refers to new group's runqueues.

In your code, I don't see where you really change task's group. AFAICS, it continues to remain in the same autogroup.

Regards,
Bharata.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Quoted 11 lines · Bharata B Rao

Quoted 2 lines

Unless I fscked up, set_task_rq() is the group change.  As soon as the task's class changes, it'll be moved to the root_task_group.

Ok, this is what I understand, may be Peter Z can confirm...

set_task_rq() just changes the task's cfs_rq and rt_rq as per its task_group(). The normal way to change a task's group is to first change its cgroup pointer (task->cgroups, see cgroup_attach_task()) After this you change the runqueues by calling set_task_rq() which now refers to new group's runqueues.

Quoted 2 lines

In your code, I don't see where you really change task's group. AFAICS, it continues to remain in the same autogroup.

IMHO, task->cgroups will not change when autogroup take effect.

Quoted 1 line · Yong Zhang

IMHO, task->cgroups will not change when autogroup take effect.

Yes and I believe this is cause for some of the weird semantics I see with autogroup and cgroups. I am not sure if this has already been discussed ealier, may be I need to go back and check the archives, but consider this:

I have cpu cgroup mounted at /cgroup and see that all the tasks in the system are listed in /cgroup/tasks file.

Now I start a task like this:

# ./while1 &
[1] 2761

and see that this task belongs to root cgroup.

# grep 2761 /cgroup/tasks
2671

But we know that this task really belongs to an autogroup and is not sitting directly on root_task_group.

# cat /proc/2761/autogroup 
/autogroup-49 nice 0

So we have a task in an autogroup (which is a sub group of root_task_group) but is being shown as part of root_task_group. Is this by design ?

Now say I create a sub cgroup and move this task to it.

# mkdir /cgroup/1
# echo 2671 > /cgroup/1/tasks

So now the task moved to a real cgroup.

# cat /cgroup/1/tasks 
2761

But the /proc/2761/autogroup isn't updated. It still shows /autogroup-49 nice 0

May be this was all discussed earlier as I noted in the beginning, but I find this semantics a bit unusual.

Quoted 12 lines · Bharata B Rao

Quoted 1 line · Yong Zhang

IMHO, task->cgroups will not change when autogroup take effect.

Yes and I believe this is cause for some of the weird semantics I see with autogroup and cgroups. I am not sure if this has already been discussed ealier, may be I need to go back and check the archives, but consider this:

I have cpu cgroup mounted at /cgroup and see that all the tasks in the system are listed in /cgroup/tasks file.

Quoted 6 lines

Now I start a task like this:

# ./while1 &
[1] 2761

and see that this task belongs to root cgroup.

Yes, cgroups sees cgroup associations, not autogroup associations.

Quoted 5 lines

# grep 2761 /cgroup/tasks
2671

But we know that this task really belongs to an autogroup and is not sitting directly on root_task_group.

Yup, cgroups doesn't want it, so autogroup (if enabled) claims it.

Quoted 5 lines

# cat /proc/2761/autogroup 
/autogroup-49 nice 0

So we have a task in an autogroup (which is a sub group of root_task_group) but is being shown as part of root_task_group. Is this by design ?

Yes, it's supposed to be transparent to cgroups.

Quoted 1 line · Mike Galbraith

Yes, it's supposed to be transparent to cgroups.

If cgroups doesn't want that task and if its transparent to cgroups, why is it shown as part of /cgroups/tasks ? The task really doesn't belong in there. Its not on the runqueue of root_task_group.

So basically I feel that a task consumed by autogroup shouldn't ideally be shown in root cgroup's tasks file.

From cgroups' viewpoint, root is where it's at. The task isn't consumed

Quoted 8 lines · Bharata B Rao

Quoted 1 line · Mike Galbraith

Yes, it's supposed to be transparent to cgroups.

If cgroups doesn't want that task and if its transparent to cgroups, why is it shown as part of /cgroups/tasks ? The task really doesn't belong in there. Its not on the runqueue of root_task_group.

Quoted 4 lines

So basically I feel that a task consumed by autogroup shouldn't ideally be shown in root cgroup's tasks file.

From cgroups' viewpoint, root is where it's at. The task isn't consumed

by autogroup either, it's only borrowed until cgroups claims it.

Would you prefer that when you mount /cgroups, you see a nearly empty root task list because box was booted with autogroup enabled?

Should
tasks appear and disappear on the fly while you're watching?

Quoted 44 lines · Mike Galbraith

Quoted 42 lines · Bharata B Rao

Quoted 23 lines · Mike Galbraith

Index: linux-2.6/kernel/sched_autogroup.c
===================================================================

kernel/sched_autogroup.c +9 -0

@@ -73,6 +73,15 @@ static inline struct autogroup *autogrou
 	ag->id = atomic_inc_return(&autogroup_seq_nr);
 	ag->tg = tg;
 	tg->autogroup = ag;
+#ifdef CONFIG_RT_GROUP_SCHED
+	/*
+	 * HACK: autogroup RT tasks run in the root task group.
+	 * This fools __sched_setscheduler() into proceeding on
+	 * so we can move the task to the appropriate runqueue
+	 * upon scheduling policy change.
+	 */
+	tg->rt_bandwidth.rt_runtime = RUNTIME_INF;
+#endif
 
 	return ag;
 

@@ -143,6 +152,15 @@ autogroup_move_group(struct task_struct
 	autogroup_kref_put(prev);
 }

kernel/sched_autogroup.c autogroup_move_group(struct task_struct · line 163

Quoted 10 lines

kernel/sched_autogroup.c +9 -0

 
+static inline void
+autogroup_setscheduler(struct task_struct *p, int on_rq)
+{
+	if (p->sched_class->task_move_group)
+		p->sched_class->task_move_group(p, on_rq);
+	else
+		set_task_rq(p, task_cpu(p));
+}
+

IIUC, with the above changes you are actually queing the task into rt_rq of an autogroup. But the task's autogroup interface (/proc/<pid>/autogroup) allows you to control the bandwidth of only cfs_rq tasks, while the rt tasks in the group get RUNTIME_INF bandwidth.

I think what we need is a real group change here (which is difficult I think) and not just sched_class change.

Quoted 2 lines

Unless I fscked up, set_task_rq() is the group change. As soon as the task's class changes, it'll be moved to the root_task_group.

Ok, this is what I understand, may be Peter Z can confirm...

set_task_rq() just changes the task's cfs_rq and rt_rq as per its task_group().

The normal way to change a task's group is to first
change its cgroup pointer (task->cgroups, see cgroup_attach_task()) After this you change the runqueues by calling set_task_rq() which now refers to new group's runqueues.

Quoted 48 lines · Bharata B Rao

Quoted 46 lines · Mike Galbraith

Quoted 40 lines · Bharata B Rao

Quoted 23 lines · Mike Galbraith

Index: linux-2.6/kernel/sched_autogroup.c
===================================================================

kernel/sched_autogroup.c +9 -0

@@ -73,6 +73,15 @@ static inline struct autogroup *autogrou
 	ag->id = atomic_inc_return(&autogroup_seq_nr);
 	ag->tg = tg;
 	tg->autogroup = ag;
+#ifdef CONFIG_RT_GROUP_SCHED
+	/*
+	 * HACK: autogroup RT tasks run in the root task group.
+	 * This fools __sched_setscheduler() into proceeding on
+	 * so we can move the task to the appropriate runqueue
+	 * upon scheduling policy change.
+	 */
+	tg->rt_bandwidth.rt_runtime = RUNTIME_INF;
+#endif
 
 	return ag;
 

@@ -143,6 +152,15 @@ autogroup_move_group(struct task_struct
 	autogroup_kref_put(prev);
 }

kernel/sched_autogroup.c autogroup_move_group(struct task_struct · line 163

Quoted 10 lines

kernel/sched_autogroup.c +9 -0

 
+static inline void
+autogroup_setscheduler(struct task_struct *p, int on_rq)
+{
+	if (p->sched_class->task_move_group)
+		p->sched_class->task_move_group(p, on_rq);
+	else
+		set_task_rq(p, task_cpu(p));
+}
+

IIUC, with the above changes you are actually queing the task into rt_rq of an autogroup. But the task's autogroup interface (/proc/<pid>/autogroup) allows you to control the bandwidth of only cfs_rq tasks, while the rt tasks in the group get RUNTIME_INF bandwidth.

Quoted 2 lines

I think what we need is a real group change here (which is difficult I think) and not just sched_class change.

Unless I fscked up, set_task_rq() is the group change. As soon as the task's class changes, it'll be moved to the root_task_group.

Quoted 4 lines

Ok, this is what I understand, may be Peter Z can confirm...

set_task_rq() just changes the task's cfs_rq and rt_rq as per its task_group().

Exactly. When you call task_group() for an RT task, it returns &root_task_group, so a task making the transition to RT will be dequeued from it's autogroup cfs_rq, and enqueued in root_task_group's rt_rq.

Quoted 40 lines · Bharata B Rao

Quoted 23 lines · Mike Galbraith

Index: linux-2.6/kernel/sched_autogroup.c
===================================================================

kernel/sched_autogroup.c +9 -0

@@ -73,6 +73,15 @@ static inline struct autogroup *autogrou
 	ag->id = atomic_inc_return(&autogroup_seq_nr);
 	ag->tg = tg;
 	tg->autogroup = ag;
+#ifdef CONFIG_RT_GROUP_SCHED
+	/*
+	 * HACK: autogroup RT tasks run in the root task group.
+	 * This fools __sched_setscheduler() into proceeding on
+	 * so we can move the task to the appropriate runqueue
+	 * upon scheduling policy change.
+	 */
+	tg->rt_bandwidth.rt_runtime = RUNTIME_INF;
+#endif
 
 	return ag;
 

@@ -143,6 +152,15 @@ autogroup_move_group(struct task_struct
 	autogroup_kref_put(prev);
 }

kernel/sched_autogroup.c autogroup_move_group(struct task_struct · line 163

Quoted 10 lines

kernel/sched_autogroup.c +9 -0

 
+static inline void
+autogroup_setscheduler(struct task_struct *p, int on_rq)
+{
+	if (p->sched_class->task_move_group)
+		p->sched_class->task_move_group(p, on_rq);
+	else
+		set_task_rq(p, task_cpu(p));
+}
+

IIUC, with the above changes you are actually queing the task into rt_rq of an autogroup. But the task's autogroup interface (/proc/<pid>/autogroup) allows you to control the bandwidth of only cfs_rq tasks, while the rt tasks in the group get RUNTIME_INF bandwidth.

Quoted 2 lines

I think what we need is a real group change here (which is difficult I think) and not just sched_class change.

Unless I fscked up, set_task_rq() is the group change. As soon as the task's class changes, it'll be moved to the root_task_group.

But it'd be better to not touch anything, just deflect RT tasks away from useless queues from the start.

sched, autogroup: fix CONFIG_RT_GROUP_SCHED sched_setscheduler() failure.

If CONFIG_RT_GROUP_SCHED is set, sched_setscheduler() fails due to autogroup not allocating rt_runtime.  Squirrel allocated but useless rt_se and rt_rq away for group destruction time, and deflect RT tasks to the root task group.

Signed-off-by Mike Galbraith <[email protected]>

Reported-by Bharata B Rao <[email protected]>

---
 kernel/sched_autogroup.c |   13 +++++++++++++
 kernel/sched_autogroup.h |    4 ++++

 2 files changed, 17 insertions(+)

Index: linux-2.6/kernel/sched_autogroup.c
===================================================================

kernel/sched_autogroup.c +5 -0

@@ -27,6 +27,11 @@ static inline void autogroup_destroy(str
  {
        struct autogroup *ag = container_of(kref, struct autogroup, kref);
 
+#ifdef CONFIG_RT_GROUP_SCHED
+       ag->tg->rt_se = ag->rt_se;
+       ag->tg->rt_rq = ag->rt_rq;
+       ag->tg->rt_bandwidth.rt_runtime = 0;
+#endif
        sched_destroy_group(ag->tg);
  }
 

Setting rt_runtime both in this patch and the previous one will make tune other's rt_runtime/rt_period trouble, I guess.

Setting rt_runtime both in this patch and the previous one will make tune other's rt_runtime/rt_period trouble, I guess.

sched, autogroup: fix CONFIG_RT_GROUP_SCHED sched_setscheduler() failure.

If CONFIG_RT_GROUP_SCHED is set, __sched_setscheduler() fails due to autogroup not allocating rt_runtime.  Free unused/unusable rt_se and rt_rq, redirect RT tasks to the root task group, and tell __sched_setscheduler() that it's ok.

Signed-off-by Mike Galbraith <[email protected]>

Reported-by Bharata B Rao <[email protected]>

kernel/sched.c +0 -0

@@ -4874,7 +4874,8 @@ recheck:
                 * assigned.
                 */

               if (rt_bandwidth_enabled() && rt_policy(policy) &&
-                               task_group(p)->rt_bandwidth.rt_runtime == 0) {
+                               task_group(p)->rt_bandwidth.rt_runtime == 0 &&

Quoted 8 lines · Mike Galbraith

sched, autogroup: fix CONFIG_RT_GROUP_SCHED sched_setscheduler() failure.

If CONFIG_RT_GROUP_SCHED is set, __sched_setscheduler() fails due to autogroup not allocating rt_runtime.  Free unused/unusable rt_se and rt_rq, redirect RT tasks to the root task group, and tell __sched_setscheduler() that it's ok.

Signed-off-by Mike Galbraith <[email protected]>

Reported-by Bharata B Rao <[email protected]>

This looks more clear ;)

And a little comment below

Quoted 10 lines

---
 kernel/sched.c           |    3 ++- kernel/sched_autogroup.c |   27 +++++++++++++++++++++++++++ kernel/sched_autogroup.h |    4 ++++

 3 files changed, 33 insertions(+), 1 deletion(-)

kernel/sched_autogroup.c +0 -0

@@ -106,6 +128,11 @@ task_wants_autogroup(struct task_struct
        return true;
  }
 

kernel/sched_autogroup.c task_wants_autogroup(struct task_struct · line 133

Quoted 3 lines

kernel/sched_autogroup.c +3 -0

+static inline bool task_group_is_autogroup(struct task_group *tg)
+{
+       return tg != &root_task_group && tg->autogroup;

Isn't just checking tg->autogroup sufficient?

if tg == &root_task_group

Quoted 8 lines

kernel/sched.c +0 -0

@@ -4874,7 +4874,8 @@ recheck:
                 * assigned.
                 */

               if (rt_bandwidth_enabled() && rt_policy(policy) &&
-                               task_group(p)->rt_bandwidth.rt_runtime == 0) {
+                               task_group(p)->rt_bandwidth.rt_runtime == 0 &&

this check will fail.

Thanks,
Yong

--
Only stand for myself

Quoted 8 lines · Mike Galbraith

sched, autogroup: fix CONFIG_RT_GROUP_SCHED sched_setscheduler() failure.

If CONFIG_RT_GROUP_SCHED is set, __sched_setscheduler() fails due to autogroup not allocating rt_runtime. Free unused/unusable rt_se and rt_rq, redirect RT tasks to the root task group, and tell __sched_setscheduler() that it's ok.

Signed-off-by Mike Galbraith <[email protected]>

Reported-by Bharata B Rao <[email protected]>

This looks more clear ;)

And a little comment below

kernel/sched_autogroup.c task_wants_autogroup(struct task_struct · line 133

Quoted 13 lines

---
 kernel/sched.c           |    3 ++-
 kernel/sched_autogroup.c |   27 +++++++++++++++++++++++++++
 kernel/sched_autogroup.h |    4 ++++
 3 files changed, 33 insertions(+), 1 deletion(-)

kernel/sched_autogroup.c +3 -0

@@ -106,6 +128,11 @@ task_wants_autogroup(struct task_struct
       return true;
 }
 
+static inline bool task_group_is_autogroup(struct task_group *tg)
+{
+       return tg != &root_task_group && tg->autogroup;

Isn't just checking tg->autogroup sufficient?

if tg == &root_task_group

kernel/sched.c recheck: · line 4877

Quoted 8 lines

kernel/sched.c +1 -1

@@ -4874,7 +4874,8 @@ recheck:
                * assigned.
                */
               if (rt_bandwidth_enabled() && rt_policy(policy) &&
-                               task_group(p)->rt_bandwidth.rt_runtime == 0) {
+                               task_group(p)->rt_bandwidth.rt_runtime == 0 &&

this check will fail.

(mailer switched me to work account again. grrrr, switch yet again:)

Quoted 2 lines · Yong Zhang

Setting rt_runtime both in this patch and the previous one will make tune other's rt_runtime/rt_period trouble, I guess.

Yeah, as in making it impossible to allocate even 1 usec :) We need zero bandwidth though, so tell __sched_setscheduler() that autogroups having zero allocated is perfectly fine.

sched, autogroup: fix CONFIG_RT_GROUP_SCHED sched_setscheduler() failure.

If CONFIG_RT_GROUP_SCHED is set, __sched_setscheduler() fails due to autogroup not allocating rt_runtime. Free unused/unusable rt_se and rt_rq, redirect RT tasks to the root task group, and tell __sched_setscheduler() that it's ok.

Signed-off-by Mike Galbraith <[email protected]>

Reported-by Bharata B Rao <[email protected]>

---
 kernel/sched.c           |    3 ++-
 kernel/sched_autogroup.c |   27 +++++++++++++++++++++++++++
 kernel/sched_autogroup.h |    4 ++++
 3 files changed, 33 insertions(+), 1 deletion(-)

sched, autogroup: fix CONFIG_RT_GROUP_SCHED sched_setscheduler() failure.

If CONFIG_RT_GROUP_SCHED is set, __sched_setscheduler() fails due to autogroup not allocating rt_runtime. Free unused/unusable rt_se and rt_rq, redirect RT tasks to the root task group, and tell __sched_setscheduler() that it's ok.

Signed-off-by Mike Galbraith <[email protected]>

Reported-by Bharata B Rao <[email protected]>

Quoted 8 lines · Mike Galbraith

sched, autogroup: fix CONFIG_RT_GROUP_SCHED sched_setscheduler() failure.

If CONFIG_RT_GROUP_SCHED is set, __sched_setscheduler() fails due to autogroup not allocating rt_runtime. Free unused/unusable rt_se and rt_rq, redirect RT tasks to the root task group, and tell __sched_setscheduler() that it's ok.

Signed-off-by Mike Galbraith <[email protected]>

Reported-by Bharata B Rao <[email protected]>

Thanks, applied!

Quoted 10 lines · Peter Zijlstra

Quoted 8 lines · Mike Galbraith

sched, autogroup: fix CONFIG_RT_GROUP_SCHED sched_setscheduler() failure.

If CONFIG_RT_GROUP_SCHED is set, __sched_setscheduler() fails due to autogroup not allocating rt_runtime. Free unused/unusable rt_se and rt_rq, redirect RT tasks to the root task group, and tell __sched_setscheduler() that it's ok.

Signed-off-by Mike Galbraith <[email protected]>

Reported-by Bharata B Rao <[email protected]>

Quoted 1 line

Thanks, applied!

while this behaviour is certeinly necessary, i think this is a hack. it fixes the problem for autogroups.
But its not fixed for things which want to control the cfs shares via normal cgroups.

why isnt rt_runtime_us residing in a separate (new) subsystem ?

Quoted 13 lines · torbenh

Quoted 11 lines · Peter Zijlstra

Quoted 8 lines · Mike Galbraith

sched, autogroup: fix CONFIG_RT_GROUP_SCHED sched_setscheduler() failure.

If CONFIG_RT_GROUP_SCHED is set, __sched_setscheduler() fails due to autogroup not allocating rt_runtime. Free unused/unusable rt_se and rt_rq, redirect RT tasks to the root task group, and tell __sched_setscheduler() that it's ok.

Signed-off-by Mike Galbraith <[email protected]>

Reported-by Bharata B Rao <[email protected]>

Thanks, applied!

Quoted 4 lines

while this behaviour is certeinly necessary, i think this is a hack. it fixes the problem for autogroups.
But its not fixed for things which want to control the cfs shares via normal cgroups.

You mean automated control ala systemd? For a static set of groups, it works fine. I was wondering how systemd would deal with it.

Quoted 2 lines

why isnt rt_runtime_us residing in a separate (new) subsystem ?

The allocation problem was shamelessly punted back to the user, where I think it truly belongs.

-Mike

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Quoted 19 lines · Mike Galbraith

Quoted 17 lines · torbenh

Quoted 11 lines · Peter Zijlstra

Quoted 8 lines · Mike Galbraith

sched, autogroup: fix CONFIG_RT_GROUP_SCHED sched_setscheduler() failure.

If CONFIG_RT_GROUP_SCHED is set, __sched_setscheduler() fails due to autogroup not allocating rt_runtime. Free unused/unusable rt_se and rt_rq, redirect RT tasks to the root task group, and tell __sched_setscheduler() that it's ok.

Signed-off-by Mike Galbraith <[email protected]>

Reported-by Bharata B Rao <[email protected]>

Thanks, applied!

while this behaviour is certeinly necessary, i think this is a hack. it fixes the problem for autogroups.
But its not fixed for things which want to control the cfs shares via normal cgroups.

Quoted 2 lines

You mean automated control ala systemd? For a static set of groups, it works fine. I was wondering how systemd would deal with it.

but i can not get the same behaviour as if CONFIG_RT_GROUP_SCHED was off. iE N cgroups with different cpu.share values, but each with rt_runtime_us=950000

if the rt_runtime_us was in a different subsystem, its my understanding that i could leave rt_runtime_us alone, and have all tasks in the root group in the rt_runtime subsystem.

Quoted 2 lines

The allocation problem was shamelessly punted back to the user, where I think it truly belongs.

sure it belongs to the user. but what if user wants to have different cpu.shares, but full rt_runtime_us for all tasks ?

Quoted 20 lines · torbenh

Quoted 19 lines · Mike Galbraith

Quoted 17 lines · torbenh

Quoted 11 lines · Peter Zijlstra

Quoted 8 lines · Mike Galbraith

sched, autogroup: fix CONFIG_RT_GROUP_SCHED sched_setscheduler() failure.

If CONFIG_RT_GROUP_SCHED is set, __sched_setscheduler() fails due to autogroup not allocating rt_runtime. Free unused/unusable rt_se and rt_rq, redirect RT tasks to the root task group, and tell __sched_setscheduler() that it's ok.

Signed-off-by Mike Galbraith <[email protected]>

Reported-by Bharata B Rao <[email protected]>

Thanks, applied!

while this behaviour is certeinly necessary, i think this is a hack. it fixes the problem for autogroups.
But its not fixed for things which want to control the cfs shares via normal cgroups.

Quoted 6 lines

Quoted 2 lines

You mean automated control ala systemd? For a static set of groups, it works fine. I was wondering how systemd would deal with it.

but i can not get the same behaviour as if CONFIG_RT_GROUP_SCHED was off. iE N cgroups with different cpu.share values, but each with rt_runtime_us=950000

? if CONFIG_RT_GROUP_SCHED was a noop, it wouldn't exist.

Quoted 3 lines

if the rt_runtime_us was in a different subsystem, its my understanding that i could leave rt_runtime_us alone, and have all tasks in the root group in the rt_runtime subsystem.

Sounds like you just want to turn CONFIG_RT_GROUP_SCHED off.

Quoted 5 lines

Quoted 2 lines

The allocation problem was shamelessly punted back to the user, where I think it truly belongs.

sure it belongs to the user. but what if user wants to have different cpu.shares, but full rt_runtime_us for all tasks ?

Then the user doesn't want CONFIG_RT_GROUP_SCHED set.

Quoted 20 lines · torbenh

Quoted 19 lines · Mike Galbraith

Quoted 17 lines · torbenh

Quoted 11 lines · Peter Zijlstra

Quoted 8 lines · Mike Galbraith

sched, autogroup: fix CONFIG_RT_GROUP_SCHED sched_setscheduler() failure.

If CONFIG_RT_GROUP_SCHED is set, __sched_setscheduler() fails due to autogroup not allocating rt_runtime. Free unused/unusable rt_se and rt_rq, redirect RT tasks to the root task group, and tell __sched_setscheduler() that it's ok.

Signed-off-by Mike Galbraith <[email protected]>

Reported-by Bharata B Rao <[email protected]>

Thanks, applied!

while this behaviour is certeinly necessary, i think this is a hack. it fixes the problem for autogroups.
But its not fixed for things which want to control the cfs shares via normal cgroups.

Quoted 6 lines

Quoted 2 lines

You mean automated control ala systemd? For a static set of groups, it works fine. I was wondering how systemd would deal with it.

but i can not get the same behaviour as if CONFIG_RT_GROUP_SCHED was off. iE N cgroups with different cpu.share values, but each with rt_runtime_us=950000

? if CONFIG_RT_GROUP_SCHED was a noop, it wouldn't exist.

Quoted 3 lines

if the rt_runtime_us was in a different subsystem, its my understanding that i could leave rt_runtime_us alone, and have all tasks in the root group in the rt_runtime subsystem.

Sounds like you just want to turn CONFIG_RT_GROUP_SCHED off.

Quoted 6 lines

Quoted 3 lines

The allocation problem was shamelessly punted back to the user, where I think it truly belongs.

sure it belongs to the user. but what if user wants to have different cpu.shares, but full rt_runtime_us for all tasks ?

Then the user doesn't want CONFIG_RT_GROUP_SCHED set.

-Mike

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Quoted 34 lines · Mike Galbraith

Quoted 20 lines · torbenh

Quoted 19 lines · Mike Galbraith

Quoted 17 lines · torbenh

Quoted 11 lines · Peter Zijlstra

Quoted 8 lines · Mike Galbraith

sched, autogroup: fix CONFIG_RT_GROUP_SCHED sched_setscheduler() failure.

If CONFIG_RT_GROUP_SCHED is set, __sched_setscheduler() fails due to autogroup not allocating rt_runtime. Free unused/unusable rt_se and rt_rq, redirect RT tasks to the root task group, and tell __sched_setscheduler() that it's ok.

Signed-off-by Mike Galbraith <[email protected]>

Reported-by Bharata B Rao <[email protected]>

Thanks, applied!

while this behaviour is certeinly necessary, i think this is a hack. it fixes the problem for autogroups.
But its not fixed for things which want to control the cfs shares via normal cgroups.

Quoted 6 lines

Quoted 2 lines

You mean automated control ala systemd? For a static set of groups, it works fine. I was wondering how systemd would deal with it.

but i can not get the same behaviour as if CONFIG_RT_GROUP_SCHED was off. iE N cgroups with different cpu.share values, but each with rt_runtime_us=950000

? if CONFIG_RT_GROUP_SCHED was a noop, it wouldn't exist.

Quoted 3 lines

if the rt_runtime_us was in a different subsystem, its my understanding that i could leave rt_runtime_us alone, and have all tasks in the root group in the rt_runtime subsystem.

Quoted 1 line

Sounds like you just want to turn CONFIG_RT_GROUP_SCHED off.

but distros turn it on.
we could prevent debian from turning it on.
now opensuse 11.4 has turned it on.

Quoted 7 lines

Quoted 5 lines

Quoted 2 lines

The allocation problem was shamelessly punted back to the user, where I think it truly belongs.

sure it belongs to the user. but what if user wants to have different cpu.shares, but full rt_runtime_us for all tasks ?

Then the user doesn't want CONFIG_RT_GROUP_SCHED set.

but distros force it onto the user.
if systemd shows up, and puts things into different cgroups, we end up with a pretty fragmented rt_runtime_us.

Quoted 37 lines · torbenh

Quoted 35 lines · Mike Galbraith

Quoted 20 lines · torbenh

Quoted 19 lines · Mike Galbraith

Quoted 17 lines · torbenh

Quoted 11 lines · Peter Zijlstra

Quoted 8 lines · Mike Galbraith

sched, autogroup: fix CONFIG_RT_GROUP_SCHED sched_setscheduler() failure.

If CONFIG_RT_GROUP_SCHED is set, __sched_setscheduler() fails due to autogroup not allocating rt_runtime. Free unused/unusable rt_se and rt_rq, redirect RT tasks to the root task group, and tell __sched_setscheduler() that it's ok.

Signed-off-by Mike Galbraith <[email protected]>

Reported-by Bharata B Rao <[email protected]>

Thanks, applied!

while this behaviour is certeinly necessary, i think this is a hack. it fixes the problem for autogroups.
But its not fixed for things which want to control the cfs shares via normal cgroups.

Quoted 6 lines

Quoted 2 lines

You mean automated control ala systemd? For a static set of groups, it works fine. I was wondering how systemd would deal with it.

but i can not get the same behaviour as if CONFIG_RT_GROUP_SCHED was off. iE N cgroups with different cpu.share values, but each with rt_runtime_us=950000

? if CONFIG_RT_GROUP_SCHED was a noop, it wouldn't exist.

Quoted 3 lines

if the rt_runtime_us was in a different subsystem, its my understanding that i could leave rt_runtime_us alone, and have all tasks in the root group in the rt_runtime subsystem.

Sounds like you just want to turn CONFIG_RT_GROUP_SCHED off.

Quoted 3 lines

but distros turn it on.
we could prevent debian from turning it on.
now opensuse 11.4 has turned it on.

If you or anyone else turns on RT_GROUP_SCHED, you will count your beans, and pay up front, or you will not play. That's a very sensible policy for realtime.

Quoted 9 lines

Quoted 8 lines

Quoted 6 lines

Quoted 3 lines

The allocation problem was shamelessly punted back to the user, where I think it truly belongs.

sure it belongs to the user. but what if user wants to have different cpu.shares, but full rt_runtime_us for all tasks ?

Then the user doesn't want CONFIG_RT_GROUP_SCHED set.

Quoted 3 lines

but distros force it onto the user.
if systemd shows up, and puts things into different cgroups, we end up with a pretty fragmented rt_runtime_us.

If systemd deals with it at all, seems to me it can only make a mess of it. But who knows, maybe they made a clever allocator. If they didn't, they'll need an escape hatch methinks.

Quoted 9 lines · Mike Galbraith

Quoted 7 lines · torbenh

Quoted 1 line · Mike Galbraith

Sounds like you just want to turn CONFIG_RT_GROUP_SCHED off.

but distros turn it on.
we could prevent debian from turning it on.
now opensuse 11.4 has turned it on.

Quoted 3 lines

If you or anyone else turns on RT_GROUP_SCHED, you will count your beans, and pay up front, or you will not play. That's a very sensible policy for realtime.

this probably means that generic computer distros should not turn this option on ?

Quoted 3 lines

If systemd deals with it at all, seems to me it can only make a mess of it. But who knows, maybe they made a clever allocator. If they didn't, they'll need an escape hatch methinks.

the problem is that audio applications can not really pre allocate their cpu needs. user can add processing plugins until he pushes his machine to the limit. (or the cgroup where his process is running in)

we dont really have a mechanism for plugins to publish their needed cycles.

Quoted 15 lines · torbenh

Quoted 12 lines · Mike Galbraith

Quoted 7 lines · torbenh

Quoted 1 line · Mike Galbraith

Sounds like you just want to turn CONFIG_RT_GROUP_SCHED off.

but distros turn it on.
we could prevent debian from turning it on.
now opensuse 11.4 has turned it on.

If you or anyone else turns on RT_GROUP_SCHED, you will count your beans, and pay up front, or you will not play. That's a very sensible policy for realtime.

Quoted 2 lines

this probably means that generic computer distros should not turn this option on ?

Yeah, agreed, not for a great default config, but only because newfangled automation thingies can't (possibly?) deal with it sanely.

Quoted 9 lines

Quoted 4 lines

If systemd deals with it at all, seems to me it can only make a mess of it. But who knows, maybe they made a clever allocator. If they didn't, they'll need an escape hatch methinks.

the problem is that audio applications can not really pre allocate their cpu needs. user can add processing plugins until he pushes his machine to the limit. (or the cgroup where his process is running in)

Quoted 2 lines

we dont really have a mechanism for plugins to publish their needed cycles.

I can't see how it could matter what any individual group of arbitrary groups N (who can appear/disappear in the blink of an eye) advertises as it's wish of the instant. "Hard" + "Arbitrary" doesn't compute for me.

-Mike

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

i dont really understnad this statement.

P.S. Kindly do not CC closed lists. Closed list readers only see your part of any conversation, and anyone replying to you is rewarded with bounce spam for their trouble.

Index: linux-2.6/kernel/sched_autogroup.c
===================================================================

kernel/sched_autogroup.c +9 -0

@@ -73,6 +73,15 @@ static inline struct autogroup *autogrou
        ag->id = atomic_inc_return(&autogroup_seq_nr);
        ag->tg = tg;
        tg->autogroup = ag;
+#ifdef CONFIG_RT_GROUP_SCHED
+       /*
+        * HACK: autogroup RT tasks run in the root task group.
+        * This fools __sched_setscheduler() into proceeding on
+        * so we can move the task to the appropriate runqueue
+        * upon scheduling policy change.
+        */
+       tg->rt_bandwidth.rt_runtime = RUNTIME_INF;
+#endif
 
        return ag;
 

Quoted 24 lines · Mike Galbraith

Index: linux-2.6/kernel/sched_autogroup.c
===================================================================

kernel/sched_autogroup.c +9 -0

@@ -73,6 +73,15 @@ static inline struct autogroup *autogrou
       ag->id = atomic_inc_return(&autogroup_seq_nr);
       ag->tg = tg;
       tg->autogroup = ag;
+#ifdef CONFIG_RT_GROUP_SCHED
+       /*
+        * HACK: autogroup RT tasks run in the root task group.
+        * This fools __sched_setscheduler() into proceeding on
+        * so we can move the task to the appropriate runqueue
+        * upon scheduling policy change.
+        */
+       tg->rt_bandwidth.rt_runtime = RUNTIME_INF;
+#endif
 
       return ag;
 

@@ -143,6 +152,15 @@ autogroup_move_group(struct task_struct
       autogroup_kref_put(prev);
 }
 

kernel/sched_autogroup.c autogroup_move_group(struct task_struct · line 159

Quoted 5 lines

kernel/sched_autogroup.c +5 -0

+static inline void
+autogroup_setscheduler(struct task_struct *p, int on_rq)
+{
+       if (p->sched_class->task_move_group)
+               p->sched_class->task_move_group(p, on_rq);

This will be called even if we don't change the sched_class in which case it is not necessary.