sched/rt: Don't try push tasks if there are none.
PATCH 1/1 sched/rt: Don't try push tasks if there are none.
I have a RT task X at a high priority and cyclictest on each CPU with lower priority than X's. If X is active and each CPU wakes their own cylictest thread then it ends in a longer rto_push storm.
A random CPU determines via balance_rt() that the CPU on which X is running needs to push tasks. X has the highest priority, cyclictest is next in line so there is nothing that can be done since the task with the higher priority is not touched.
tell_cpu_to_push() increments rto_loop_next and schedules rto_push_irq_work_func() on X's CPU. The other CPUs also increment the loop counter and do the same. Once rto_push_irq_work_func() is active it does nothing because it has _no_ pushable tasks on its runqueue. Then checks rto_next_cpu() and decides to queue irq_work on the local CPU because another CPU requested a push by incrementing the counter.
I have traces where ~30 CPUs request this ~3 times each before it finally ends. This greatly increases X's runtime while X isn't making much progress.
Teach rto_next_cpu() to only return CPUs which also have tasks on their runqueue which can be pushed away. This does not reduce the tell_cpu_to_push() invocations (rto_loop_next counter increments) but reduces the amount of issued rto_push_irq_work_func() if nothing can be done. As the result the overloaded CPU is blocked less often.
There are still cases where the "same job" is repeated several times (for instance the current CPU needs to resched but didn't yet because the irq-work is repeated a few times and so the old task remains on the CPU) but the majority of request end in tell_cpu_to_push() before an IPI is issued.
--- kernel/sched/rt.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
kernel/sched/rt.c +4 -1
@@ -2247,8 +2247,11 @@ static int rto_next_cpu(struct root_domain *rd) rd->rto_cpu = cpu; -if (cpu < nr_cpu_ids) +if (cpu < nr_cpu_ids) { +if (!has_pushable_tasks (cpu_rq (cpu))) +continue ;return cpu; + } rd->rto_cpu =-1 ;
Signature · 1 line
2.40.1
kernel/sched/rt.c static int rto_next_cpu(struct root_domain *rd) · line 2252
Quoted 12 lines · On Tue, 1 Aug 2023 17:26:48 +0200 Sebastian Andrzej Siewior…
kernel/sched/rt.c +3
-1@@ -2247,8 +2247,11 @@ static int rto_next_cpu(struct root_domain *rd) rd->rto_cpu = cpu; -if (cpu < nr_cpu_ids) +if (cpu < nr_cpu_ids) { +if (!has_pushable_tasks (cpu_rq (cpu))) +continue ;
Hmm, I had to read this again to make sure we can't get into an infinite loop. But it looks like we are safe, as if none of the CPUs had pushable tasks, we would still hit the cpu == nr_cpu_ids case and fall into the loop check, and break out there.
-- Steve
Quoted 5 lines
kernel/sched/rt.c +1
-0return cpu; + } rd->rto_cpu =-1 ;
Valentin Schneider
reviewerQuoted 8 lines · Sebastian Andrzej Siewior
I have a RT task X at a high priority and cyclictest on each CPU with lower priority than X's. If X is active and each CPU wakes their own cylictest thread then it ends in a longer
rto_pushstorm.
A random CPU determines viabalance_rt()that the CPU on which X is running needs to push tasks. X has the highest priority, cyclictest is next in line so there is nothing that can be done since the task with the higher priority is not touched.
Quoted 7 lines
tell_cpu_to_push()incrementsrto_loop_nextand schedulesrto_push_irq_work_func()on X's CPU. The other CPUs also increment the loop counter and do the same. Oncerto_push_irq_work_func()is active it does nothing because it has _no_ pushable tasks on its runqueue. Then checksrto_next_cpu()and decides to queueirq_workon the local CPU because another CPU requested a push by incrementing the counter.
For a CPU to be in the rto_mask, it needs:
rt_rq->rt_nr_migratory && rt_rq->rt_nr_total > 1
But if that CPU has no pushable tasks, then that means only the current task has p->nr_cpus_allowed > 1.
Should we change it so a CPU is only in the rto_mask iff it has pushable tasks? AFAICT that should not break the case where we push the current task away due to migration_disabled, as that still relies on the migration_disabled task to be in the pushable list.
Quoted 21 lines · Valentin Schneider
Quoted 8 lines · Sebastian Andrzej Siewior
I have a RT task X at a high priority and cyclictest on each CPU with lower priority than X's. If X is active and each CPU wakes their own cylictest thread then it ends in a longer
rto_pushstorm.
A random CPU determines viabalance_rt()that the CPU on which X is running needs to push tasks. X has the highest priority, cyclictest is next in line so there is nothing that can be done since the task with the higher priority is not touched.Quoted 7 lines
tell_cpu_to_push()incrementsrto_loop_nextand schedulesrto_push_irq_work_func()on X's CPU. The other CPUs also increment the loop counter and do the same. Oncerto_push_irq_work_func()is active it does nothing because it has _no_ pushable tasks on its runqueue. Then checksrto_next_cpu()and decides to queueirq_workon the local CPU because another CPU requested a push by incrementing the counter.For a CPU to be in the
rto_mask, it needs:rt_rq->rt_nr_migratory && rt_rq->rt_nr_total > 1
Quoted 7 lines
But if that CPU has no pushable tasks, then that means only the current task has
p->nr_cpus_allowed> 1.Should we change it so a CPU is only in the
rto_maskiff it has pushable tasks? AFAICT that should not break the case where we push the current task away due tomigration_disabled, as that still relies on themigration_disabledtask to be in the pushable list.
Sounds good. The task with the highest priority becomes pushable if it gets preempted (by a task with higher priority). This gets considered, right?
Sebastian
Valentin Schneider
reviewerQuoted 25 lines · Sebastian Andrzej Siewior
Quoted 24 lines · Valentin Schneider
Quoted 8 lines · Sebastian Andrzej Siewior
I have a RT task X at a high priority and cyclictest on each CPU with lower priority than X's. If X is active and each CPU wakes their own cylictest thread then it ends in a longer
rto_pushstorm.
A random CPU determines viabalance_rt()that the CPU on which X is running needs to push tasks. X has the highest priority, cyclictest is next in line so there is nothing that can be done since the task with the higher priority is not touched.Quoted 7 lines
tell_cpu_to_push()incrementsrto_loop_nextand schedulesrto_push_irq_work_func()on X's CPU. The other CPUs also increment the loop counter and do the same. Oncerto_push_irq_work_func()is active it does nothing because it has _no_ pushable tasks on its runqueue. Then checksrto_next_cpu()and decides to queueirq_workon the local CPU because another CPU requested a push by incrementing the counter.For a CPU to be in the
rto_mask, it needs:rt_rq->rt_nr_migratory && rt_rq->rt_nr_total > 1But if that CPU has no pushable tasks, then that means only the current task has
p->nr_cpus_allowed> 1.
Quoted 9 lines
Quoted 4 lines
Should we change it so a CPU is only in the
rto_maskiff it has pushable tasks? AFAICT that should not break the case where we push the current task away due tomigration_disabled, as that still relies on themigration_disabledtask to be in the pushable list.Sounds good. The task with the highest priority becomes pushable if it gets preempted (by a task with higher priority). This gets considered, right?
Yeah, when we switch the current task in put_prev_task_rt() we add the previously-running (IOW preempted) task to the pushable list (if isn't affined to just one CPU).
Now it looks like if we change update_rt_migration() to look at the pushable list instead of rt_nr_migratory, we could get rid of rt_nr_migratory itself - let me poke at this a little, I might just be about to figure out why that isn't already the case...
Quoted 1 line
Sebastian