Skip to contentSkip to conversation

1 RT task blocks 4-core machine ?

DISCUSSION Tommaso Cucinotta 2 participants 2 messages Oct 9, 2010 – Oct 11, 2010

lore
Share & tools
lore
#0 lore raw
Quoted 11 lines · Peter
Quoted 6 lines · Tommaso Cucinotta

A possible explanation might be that the CFS load balancing logic sees my only active task (e.g., the ssh server or shell etc.) as running alone on its core, and does not detect that it is inhibited to actually run due to RT tasks on the same core. Therefore, it will not migrate the task to the free cores. Does this explanation make sense or is it completely wrong ?

Possibly, its got some logic to detect this but maybe it gets confused still, in particular look at the adaptive cpu_power in update_cpu_power() and calling functions.

Ok, I'll have a look (when I have some time :-( ), thanks.

Quoted 7 lines
Quoted 2 lines

Also, I'd like to hear whether this is considered the "normal/desired" behavior of intermixing RT and non-RT tasks.

Pegging a cpu using sched_fifo/rr pretty much means you get to keep the pieces, if it works nice, if you can make it work better kudos, but no polling from sched_fifo/rr is not something that is considered sane for the general health of your system.

Sure, I was not thinking to push/pull across heterogeneous scheduling classes, but rather to simply account for the proper per-CPU tasks count and load (including all the tasks comprising RT ones) when load-balancing in CFS. Perhaps, you mean, e.g., if a RT task ends, the CPU would go idle and it would be supposed to pull ? Just we don't do that, and at the next load-balancing decision things would be fixed up (please, consider I don't know the CFS load balancer so well).

So, for example, in addition to fix the reported issue, we'd get also that, when pinning a heavy RT workload on a CPU, CFS tasks would migrate to other CPUs, if available. Again, that doesn't need to be instantaneous (push), but it could happen later when the CFS load-balancer is invoked (is it invoked periodically, as of now ?).

Thanks,

	T.
Signature · 4 lines
Tommaso Cucinotta, Computer Engineering PhD, Researcher
ReTiS Lab, Scuola Superiore Sant'Anna, Pisa, Italy
Tel +39 050 882 024, Fax +39 050 882 003
http://retis.sssup.it/people/tommaso
#1 lore raw
Quoted 23 lines · Tommaso Cucinotta
Quoted 11 lines · Peter
Quoted 6 lines · Tommaso Cucinotta

A possible explanation might be that the CFS load balancing logic sees my only active task (e.g., the ssh server or shell etc.) as running alone on its core, and does not detect that it is inhibited to actually run due to RT tasks on the same core. Therefore, it will not migrate the task to the free cores. Does this explanation make sense or is it completely wrong ?

Possibly, its got some logic to detect this but maybe it gets confused still, in particular look at the adaptive cpu_power in update_cpu_power() and calling functions.

Ok, I'll have a look (when I have some time :-( ), thanks.

Quoted 7 lines
Quoted 2 lines

Also, I'd like to hear whether this is considered the "normal/desired" behavior of intermixing RT and non-RT tasks.

Pegging a cpu using sched_fifo/rr pretty much means you get to keep the pieces, if it works nice, if you can make it work better kudos, but no polling from sched_fifo/rr is not something that is considered sane for the general health of your system.

Quoted 4 lines

Sure, I was not thinking to push/pull across heterogeneous scheduling classes, but rather to simply account for the proper per-CPU tasks count and load (including all the tasks comprising RT ones) when load-balancing in CFS.

Right, so we do that. Part of the problem is that RR/FIFO tasks have no weight/load (not even a worst case weight like sporadic tasks have). So what we do is (per-cpu) take an average measure of the time spend on ! CFS tasks (sched_rt_avg_update() and friends) and use that to lower that CPUs total throughput, which is reflected in the mentioned ->cpu_power variable.

Quoted 4 lines

Perhaps, you mean, e.g., if a RT task ends, the CPU would go idle and it would be supposed to pull ? Just we don't do that, and at the next load-balancing decision things would be fixed up (please, consider I don't know the CFS load balancer so well).

No, what I meant was that if a particular CPU is very busy with !CFS work, its ->cpu_power variable will decrease to 1 (0 will get us division by zero issues). Somehow we need to avoid this load-balancer from thinking its a good idea to place tasks there.

The natural balance is to move tasks away from weak CPUs, but clearly its not good enough.

Also, there is housekeeping that needs to be done on a per-cpu basis. CPU affine tasks like workqueue things need to run in order to keep the system functional, pegging a CPU with a RT task starves these, causing general system dysfunction.

Quoted 5 lines

So, for example, in addition to fix the reported issue, we'd get also that, when pinning a heavy RT workload on a CPU, CFS tasks would migrate to other CPUs, if available. Again, that doesn't need to be instantaneous (push), but it could happen later when the CFS load-balancer is invoked (is it invoked periodically, as of now ?).

That should basically work, we normalize the cpu load (sum of all cfs task weights) by the ->cpu_power, a weak cpu will tend to get all its tasks migrated away to stronger CPUs, again, there's probably some corner case that doesn't quite work as expected.