sched/rt: fix SCHED_RR across cgroups
PATCH 1/1 sched/rt: fix SCHED_RR across cgroups
task_tick_rt has an optimization to only reschedule SCHED_RR tasks if they were the only element on their rq. However, with cgroups a SCHED_RR task could be the only element on its per-cgroup rq but still be competing with other SCHED_RR tasks in its parent's cgroup. In this case, the SCHED_RR task in the child cgroup would never yield at the end of its timeslice. If the child cgroup rt_runtime_us was the same as the parent cgroup rt_runtime_us, the task in the parent cgroup would starve completely.
Modify task_tick_rt to check that the task is the only task on its rq, and that the each of the scheduling entities of its ancestors is also the only entity on its rq.
--- kernel/sched/rt.c | 15 ++++++++++----- 1 files changed, 10 insertions(+), 5 deletions(-)
kernel/sched/rt.c +10 -5
@@ -1983,6 +1983,8 @@ static void watchdog(struct rq *rq, struct task_struct *p)static void task_tick_rt (struct rq *rq,struct task_struct *p,int queued) { +struct sched_rt_entity *rt_se = &p->rt; +update_curr_rt (rq);watchdog (rq, p);
@@ -2000,12 +2002,15 @@ static void task_tick_rt(struct rq *rq, struct task_struct *p, int queued) p->rt.time_slice =RR_TIMESLICE ;/* -* Requeue to the end of queue if we are not the only element -* on the queue: +* Requeue to the end of queue if we (and all of our ancestors) are the +* only element on the queue */ -if (p->rt.run_list.prev != p->rt.run_list.next) { -requeue_task_rt (rq, p,0 ); -set_tsk_need_resched (p); +for_each_sched_rt_entity (rt_se) { +if (rt_se->run_list.prev != rt_se->run_list.next) { +requeue_task_rt (rq, p,0 ); +set_tsk_need_resched (p); +return ; + } } }
Signature · 1 line
1.7.7.3
Quoted 9 lines · Colin Cross
task_tick_rthas an optimization to only rescheduleSCHED_RRtasks if they were the only element on their rq. However, with cgroups aSCHED_RRtask could be the only element on its per-cgroup rq but still be competing with otherSCHED_RRtasks in its parent's cgroup. In this case, theSCHED_RRtask in the child cgroup would never yield at the end of its timeslice. If the child cgrouprt_runtime_uswas the same as the parent cgrouprt_runtime_us, the task in the parent cgroup would starve completely.
Quoted 5 lines
Modify
task_tick_rtto check that the task is the only task on its rq, and that the each of the scheduling entities of its ancestors is also the only entity on its rq.Signed-off-by Colin Cross <[email protected]>
OK, fair enough.. one does wonder though, WTH is android doing with SCHED_RR?
Quoted 10 lines · Peter Zijlstra
Quoted 9 lines · Colin Cross
task_tick_rthas an optimization to only rescheduleSCHED_RRtasks if they were the only element on their rq. However, with cgroups aSCHED_RRtask could be the only element on its per-cgroup rq but still be competing with otherSCHED_RRtasks in its parent's cgroup. In this case, theSCHED_RRtask in the child cgroup would never yield at the end of its timeslice. If the child cgrouprt_runtime_uswas the same as the parent cgrouprt_runtime_us, the task in the parent cgroup would starve completely.
Quoted 8 lines
Quoted 5 lines
Modify
task_tick_rtto check that the task is the only task on its rq, and that the each of the scheduling entities of its ancestors is also the only entity on its rq.Signed-off-by Colin Cross <[email protected]>OK, fair enough.. one does wonder though, WTH is android doing with
SCHED_RR?
Nothing, I was just experimenting with how it interacted with cgroups and the numbers didn't make sense.
Quoted 5 lines · Colin Cross
Quoted 2 lines
OK, fair enough.. one does wonder though, WTH is android doing with
SCHED_RR?Nothing, I was just experimenting with how it interacted with cgroups and the numbers didn't make sense.
OK. Thanks anyway!
Quoted 9 lines · Peter Zijlstra
Quoted 5 lines · Colin Cross
Quoted 2 lines
OK, fair enough.. one does wonder though, WTH is android doing with
SCHED_RR?Nothing, I was just experimenting with how it interacted with cgroups and the numbers didn't make sense.
OK. Thanks anyway!
Even with this patch, scheduling of SCHED_RR tasks in cgroups is a little odd. Each cgroup is treated as a schedulable entity alongside the tasks in the same parent cgroup, and then the tasks inside the child cgroup round robin through the child cgroup's time slices. So in the setup:root_cgroup
task 1
cgroup
task 2
task 3
The RR will be:
task 1, cgroup(task 2), task 1, cgroup(task 3), ...
task 1 will run twice as often, for a full RR_TIMESLICE each time, as tasks 2 and 3.
Is that the way SCHED_RR is intended to interact with cgroups?
Quoted 11 lines · Colin Cross
Even with this patch, scheduling of
SCHED_RRtasks in cgroups is a little odd. Each cgroup is treated as a schedulable entity alongside the tasks in the same parent cgroup, and then the tasks inside the child cgroup round robin through the child cgroup's time slices. So in the setup:root_cgroup
task 1
cgroup
task 2
task 3
Quoted 6 lines
The RR will be:
task 1, cgroup(task 2), task 1, cgroup(task 3), ...task 1 will run twice as often, for a full
RR_TIMESLICEeach time, as tasks 2 and 3.
That looks right to me...
Quoted 2 lines
Is that the way
SCHED_RRis intended to interact with cgroups?
I would say it is. That's what you get because of putting task1 and cgroup at the same level in the "hierarchy". I'm curious, what kind of behaviour were you expecting?
Of course, the actual schedule also depends on the real-time priority of the various tasks (groups don't have a priority, they inherit it from their tasks, or at least it was like this when I used to work with it :-P), but I guess you're putting all the tasks in the same queue (i.e., same rt-prio), is it that the case?
Dario
Quoted 5 lines
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Signature · 9 lines
<<This happens because I choose it to happen!>> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://retis.sssup.it/people/faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) -- <<This happens because I choose it to happen!>> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://retis.sssup.it/people/faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)
Quoted 18 lines · Dario Faggioli
Quoted 11 lines · Colin Cross
Even with this patch, scheduling of
SCHED_RRtasks in cgroups is a little odd. Each cgroup is treated as a schedulable entity alongside the tasks in the same parent cgroup, and then the tasks inside the child cgroup round robin through the child cgroup's time slices. So in the setup:root_cgroup
task 1
cgroup
task 2
task 3Quoted 6 lines
The RR will be:
task 1, cgroup(task 2), task 1, cgroup(task 3), ...task 1 will run twice as often, for a full
RR_TIMESLICEeach time, as tasks 2 and 3.
Quoted 7 lines
That looks right to me...
Quoted 2 lines
Is that the way
SCHED_RRis intended to interact with cgroups?I would say it is. That's what you get because of putting task1 and cgroup at the same level in the "hierarchy". I'm curious, what kind of behaviour were you expecting?
That behavior matches exactly with scheduling of normal tasks and cgroups with default cpu.shares, but doesn't match too well with what I can see of the posix SCHED_RR description, which suggests all the SCHED_RR threads go into a single queue. I was just curious if the behavior my patch restored was correct, since it can't be adjusted by tweaking any parameters like cpu.shares.
Quoted 5 lines
Of course, the actual schedule also depends on the real-time priority of the various tasks (groups don't have a priority, they inherit it from their tasks, or at least it was like this when I used to work with it :-P), but I guess you're putting all the tasks in the same queue (i.e., same rt-prio), is it that the case?
Yes.
Quoted 11 lines · Colin Cross
Quoted 3 lines
I would say it is. That's what you get because of putting task1 and cgroup at the same level in the "hierarchy". I'm curious, what kind of behaviour were you expecting?
That behavior matches exactly with scheduling of normal tasks and cgroups with default cpu.shares, but doesn't match too well with what I can see of the posix
SCHED_RRdescription, which suggests all theSCHED_RRthreads go into a single queue. I was just curious if the behavior my patch restored was correct, since it can't be adjusted by tweaking any parameters like cpu.shares.
Again, I really think it is the intended behaviour, and yes, real-time group scheduling "breaks" the POSIX specification of the SCHED_{FIFO,RR} policies intentionally (and _proudly_, as Peter would say it, am I wrong? :-P).
Dario
Signature · 4 lines
<<This happens because I choose it to happen!>> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://retis.sssup.it/people/faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)
Quoted 4 lines · Dario Faggioli
Again, I really think it is the intended behaviour, and yes, real-time group scheduling "breaks" the POSIX specification of the SCHED_{FIFO,RR} policies intentionally (and _proudly_, as Peter would say it, am I wrong? :-P).
No, cgroups are well outside of POSIX ;-) as is SMP in fact.
Steven Rostedt
reviewerQuoted 7 lines · Peter Zijlstra
Quoted 4 lines · Dario Faggioli
Again, I really think it is the intended behaviour, and yes, real-time group scheduling "breaks" the POSIX specification of the SCHED_{FIFO,RR} policies intentionally (and _proudly_, as Peter would say it, am I wrong? :-P).
No, cgroups are well outside of POSIX ;-) as is SMP in fact.
That's because the POSIX standards committee is still struggling to come up with standardized SMP calls to handle NR_CPUS = 0
Isn't Paul on that committee? ;-)
-- Steve
Quoted 11 lines · Steven Rostedt
Quoted 7 lines · Peter Zijlstra
Quoted 4 lines · Dario Faggioli
Again, I really think it is the intended behaviour, and yes, real-time group scheduling "breaks" the POSIX specification of the SCHED_{FIFO,RR} policies intentionally (and _proudly_, as Peter would say it, am I wrong? :-P).
No, cgroups are well outside of POSIX ;-) as is SMP in fact.
That's because the POSIX standards committee is still struggling to come up with standardized SMP calls to handle
NR_CPUS= 0
;-) ;-) ;-)
Quoted 1 line
Isn't Paul on that committee? ;-)
I have met with them occasionally, but have spent most of my time on the C/C++ committees. I didn't try sounding them out on NR_CPUS=0, partly because they were choking pretty hard on NR_CPUS=4096.
At least part of the problem is that every OS out there has different SMP feature, so the only way it would be possible to get this sort of thing through the committee would be to invent something that was roughly equally incompatible with everyone.
However, there are some SMP features standardized by various random committees and aggregated by The Open Group, including:
o The pthread_mutex_lock() API
o pthread_getspecific() and pthread_setspecific()
o pthread_getconcurrency() and pthread_setconcurrency()
But yes, even the aggregated standard is quite limiting.
Thanx, Paul
[tip:sched/urgent] sched/rt: Fix SCHED_RR across cgroups
In reply to Colin CrossCommit-ID: 454c79999f7eaedcdf4c15c449e43902980cbdf5 Gitweb: http://git.kernel.org/tip/454c79999f7eaedcdf4c15c449e43902980cbdf5 Author: Colin Cross <[email protected]> AuthorDate: Wed, 16 May 2012 21:34:23 -0700 Committer: Ingo Molnar <[email protected]> CommitDate: Wed, 30 May 2012 14:02:25 +0200
sched/rt: Fix SCHED_RR across cgroups
task_tick_rt() has an optimization to only reschedule SCHED_RR tasks if they were the only element on their rq. However, with cgroups a SCHED_RR task could be the only element on its per-cgroup rq but still be competing with other SCHED_RR tasks in its parent's cgroup. In this case, the SCHED_RR task in the child cgroup would never yield at the end of its timeslice. If the child cgroup rt_runtime_us was the same as the parent cgroup rt_runtime_us, the task in the parent cgroup would starve completely.
Modify task_tick_rt() to check that the task is the only task on its rq, and that the each of the scheduling entities of its ancestors is also the only entity on its rq.
--- kernel/sched/rt.c | 15 ++++++++++----- 1 files changed, 10 insertions(+), 5 deletions(-)
kernel/sched/rt.c +10 -5
@@ -1985,6 +1985,8 @@ static void watchdog(struct rq *rq, struct task_struct *p)static void task_tick_rt (struct rq *rq,struct task_struct *p,int queued) { +struct sched_rt_entity *rt_se = &p->rt; +update_curr_rt (rq);watchdog (rq, p);
@@ -2002,12 +2004,15 @@ static void task_tick_rt(struct rq *rq, struct task_struct *p, int queued) p->rt.time_slice =RR_TIMESLICE ;/* -* Requeue to the end of queue if we are not the only element -* on the queue: +* Requeue to the end of queue if we (and all of our ancestors) are the +* only element on the queue */ -if (p->rt.run_list.prev != p->rt.run_list.next) { -requeue_task_rt (rq, p,0 ); -set_tsk_need_resched (p); +for_each_sched_rt_entity (rt_se) { +if (rt_se->run_list.prev != rt_se->run_list.next) { +requeue_task_rt (rq, p,0 ); +set_tsk_need_resched (p); +return ; + } } }