RT scheduling: wakeup bug?
I've been trying to track down some unexpected realtime latencies and believe one source is a bug in the wakeup code. Specifically, this is within the try_to_wake_up() routine. Within this routine there is the following code segment:
/*
* If a newly woken up RT task cannot preempt the
* current (RT) task (on a target runqueue) then try
* to find another CPU it can preempt:
*/
if (rt_task(p) && !TASK_PREEMPTS_CURR(p, rq)) {
struct rq *this_rq = cpu_rq(this_cpu);
/*
* Special-case: the task on this CPU can be
* preempted. In that case there's no need to
* trigger reschedules on other CPUs, we can
* mark the current task for reschedule.
*
* (Note that it's safe to access this_rq without
* extra locking in this particular case, because
* we are on the current CPU.)
*/
if (TASK_PREEMPTS_CURR(p, this_rq))
set_tsk_need_resched(this_rq->curr);
else
/*
* Neither the intended target runqueue
* nor the current CPU can take this task.
* Trigger a reschedule on all other CPUs
* nevertheless, maybe one of them can take
* this task:
*/
smp_send_reschedule_allbutself_cpumask(p->cpus_allowed);
schedstat_inc(this_rq, rto_wakeup);
}
This logic seems appropriate. But, the task 'p' is most likely not on the runqueue when sending the IPI. It gets added to the runqueue a little later in the routine. As a result, the 'rt_overload' global may not be set (based on the count of RT tasks on the runqueue) and other CPUs may 'pass over' the runqueue when doing RT load balancing.
My observations/debugging/conclusions are based on an earlier version of the code. It appears the same code/issue still exists in the most version. But, I have not not done any work with the latest version.
Signature · 1 line
Mike
Re: -rt scheduling: wakeup bug?
hi Mike,
Quoted 43 lines · Mike Kravetz
I've been trying to track down some unexpected realtime latencies and believe one source is a bug in the wakeup code. Specifically, this is within the
try_to_wake_up()routine. Within this routine there is the following code segment:/*
* If a newly woken up RT task cannot preempt the
* current (RT) task (on a target runqueue) then try
* to find another CPU it can preempt:
*/
if (rt_task(p)&& !TASK_PREEMPTS_CURR(p, rq)) {
struct rq*this_rq=cpu_rq(this_cpu);
/*
* Special-case: the task on this CPU can be
* preempted. In that case there's no need to
* trigger reschedules on other CPUs, we can
* mark the current task for reschedule.
*
* (Note that it's safe to accessthis_rqwithout
* extra locking in this particular case, because
* we are on the current CPU.)
*/
if (TASK_PREEMPTS_CURR(p, this_rq))
set_tsk_need_resched(this_rq->curr);
else
/*
* Neither the intended target runqueue
* nor the current CPU can take this task.
* Trigger a reschedule on all other CPUs
* nevertheless, maybe one of them can take
* this task:
*/
smp_send_reschedule_allbutself_cpumask(p->cpus_allowed);
schedstat_inc(this_rq, rto_wakeup);
}This logic seems appropriate. But, the task 'p' is most likely not on the runqueue when sending the IPI. It gets added to the runqueue a little later in the routine. As a result, the '
rt_overload' global may not be set (based on the count of RT tasks on the runqueue) and other CPUs may 'pass over' the runqueue when doing RT load balancing.
Quoted 3 lines
My observations/debugging/conclusions are based on an earlier version of the code. It appears the same code/issue still exists in the most version. But, I have not not done any work with the latest version.
I believe you are right - nice catch of this very nontrivial bug! The patch below is against .23-rc - do you think this fix (of moving the rt wakeup sequence to after the activate_task()) is adequate?
Ingo
kernel/sched.c +22 -22
@@ -1819,6 +1819,13 @@ out_set_cpu: cpu =task_cpu (p); } +out_activate: +#endif /* CONFIG_SMP */ + +activate_task (rq, p,1 ); + +trace_start_sched_wakeup (p, rq); +/* * If a newly woken up RT task cannot preempt the * current (RT) task (on a target runqueue) then try
@@ -1849,28 +1856,21 @@ out_set_cpu:smp_send_reschedule_allbutself_cpumask (p->cpus_allowed);schedstat_inc (this_rq, rto_wakeup); - } - -out_activate: -#endif /* CONFIG_SMP */ - -activate_task (rq, p,1 ); - -trace_start_sched_wakeup (p, rq); - -/* -* Sync wakeups (i.e. those types of wakeups where the waker -* has indicated that it will leave the CPU in short order) -* don't trigger a preemption, if the woken up task will run on -* this cpu. (in this case the 'I will reschedule' promise of -* the waker guarantees that the freshly woken up task is going -* to be considered on this CPU.) -*/ -if (!sync || cpu != this_cpu) -check_preempt_curr (rq, p); -else { -if (TASK_PREEMPTS_CURR (p, rq)) -set_tsk_need_resched_delayed (rq->curr); + }else { +/* +* Sync wakeups (i.e. those types of wakeups where the waker +* has indicated that it will leave the CPU in short order) +* don't trigger a preemption, if the woken up task will run on +* this cpu. (in this case the 'I will reschedule' promise of +* the waker guarantees that the freshly woken up task is going +* to be considered on this CPU.) +*/ +if (!sync || cpu != this_cpu) +check_preempt_curr (rq, p); +else { +if (TASK_PREEMPTS_CURR (p, rq)) +set_tsk_need_resched_delayed (rq->curr); + } }if (rq->curr && p && rq &&_need_resched ())trace_special_pid (p->pid,PRIO (p),PRIO (rq->curr));
Re: -rt scheduling: wakeup bug?
Quoted 9 lines · Ingo Molnar
Quoted 4 lines · Mike Kravetz
My observations/debugging/conclusions are based on an earlier version of the code. It appears the same code/issue still exists in the most version. But, I have not not done any work with the latest version.
I believe you are right - nice catch of this very nontrivial bug! The patch below is against .23-rc - do you think this fix (of moving the rt wakeup sequence to after the
activate_task()) is adequate?
Yes, I have been running with a similar patch on a (much) earlier version of the code. It has helped quite a bit. I would have put together a patch for a later version, but my test environment is limited to this earlier version.
Signature · 1 line
Mike
Re: -rt scheduling: wakeup bug?
--
Quoted 7 lines · Mike Kravetz
Quoted 6 lines · Ingo Molnar
Quoted 4 lines · Mike Kravetz
My observations/debugging/conclusions are based on an earlier version of the code. It appears the same code/issue still exists in the most version. But, I have not not done any work with the latest version.
Quoted 8 lines
Quoted 3 lines
I believe you are right - nice catch of this very nontrivial bug! The patch below is against .23-rc - do you think this fix (of moving the rt wakeup sequence to after the
activate_task()) is adequate?Yes, I have been running with a similar patch on a (much) earlier version of the code. It has helped quite a bit. I would have put together a patch for a later version, but my test environment is limited to this earlier version.
FYI,
I've incorporated Ingo's patch into the lastest -rt release.
http://www.kernel.org/pub/linux/kernel/projects/rt/patch-2.6.23-rc9-rt1.bz2
-- Steve
Re: -rt scheduling: wakeup bug?
In reply to Ingo MolnarQuoted 52 lines · Ingo Molnar
Index:
linux-rt-rebase.q/kernel/sched.c
===================================================================kernel/sched.c +15
-22@@ -1819,6 +1819,13 @@ out_set_cpu: cpu =task_cpu (p); } +out_activate: +#endif /* CONFIG_SMP */ + +activate_task (rq, p,1 ); + +trace_start_sched_wakeup (p, rq); +/* * If a newly woken up RT task cannot preempt the * current (RT) task (on a target runqueue) then try @@ -1849,28 +1856,21 @@ out_set_cpu:smp_send_reschedule_allbutself_cpumask (p->cpus_allowed);schedstat_inc (this_rq, rto_wakeup); - } - -out_activate: -#endif /* CONFIG_SMP */ - -activate_task (rq, p,1 ); - -trace_start_sched_wakeup (p, rq); - -/* -* Sync wakeups (i.e. those types of wakeups where the waker -* has indicated that it will leave the CPU in short order) -* don't trigger a preemption, if the woken up task will run on -* this cpu. (in this case the 'I will reschedule' promise of -* the waker guarantees that the freshly woken up task is going -* to be considered on this CPU.) -*/ -if (!sync || cpu != this_cpu) -check_preempt_curr (rq, p); -else { -if (TASK_PREEMPTS_CURR (p, rq)) -set_tsk_need_resched_delayed (rq->curr); + }else { +/* +* Sync wakeups (i.e. those types of wakeups where the waker +* has indicated that it will leave the CPU in short order) +* don't trigger a preemption, if the woken up task will run on +* this cpu. (in this case the 'I will reschedule' promise of +* the waker guarantees that the freshly woken up task is going +* to be considered on this CPU.)
kernel/sched.c out_set_cpu: · line 1876
Quoted 10 lines
kernel/sched.c +7
-0+*/ +if (!sync || cpu != this_cpu) +check_preempt_curr (rq, p); +else { +if (TASK_PREEMPTS_CURR (p, rq)) +set_tsk_need_resched_delayed (rq->curr); + } }if (rq->curr && p && rq &&_need_resched ())trace_special_pid (p->pid,PRIO (p),PRIO (rq->curr));
Not an issue with the patch, just that last bit of code pulled in for context. I don't think it is a bug, but the checking of 'rq' after checking 'rq->curr' just doesn't look right (or necessary). Could it just be an artifact from earlier versions of the code?
Signature · 1 line
Mike
Re: -rt scheduling: wakeup bug?
Quoted 7 lines · Mike Kravetz
kernel/sched.c out_set_cpu: · line 1876
Quoted 2 lines
kernel/sched.c +0
-0if (rq->curr && p && rq &&_need_resched ())trace_special_pid (p->pid,PRIO (p),PRIO (rq->curr));Not an issue with the patch, just that last bit of code pulled in for context. I don't think it is a bug, but the checking of 'rq' after checking '
rq->curr' just doesn't look right (or necessary). Could it just be an artifact from earlier versions of the code?
yeah, you are right - and rq shouldnt ever be NULL there anyway.
Ingo