sched/cache: Fix a thread aggregation conflict when there is one runnable task
PATCH 1/1 sched/cache: Fix a thread aggregation conflict when there is one runnable task
On systems with SD_ASYM_PACKING set at the PKG domain level (e.g., with ITMT enabled), the sched_asym() check in sched_balance_find_src_rq() prevents pulling a single task from a higher-priority CPU to a lower-priority CPU. This blocks migrate_llc_task migrations where a task wants to move to its preferred LLC for cache locality.
For example, CPU 0 (highest priority) has a task preferring LLC3, but the load balancer on CPU 120 (LLC3) cannot pull it because:sched_asym(PKG_sd, cpu=0, dst_cpu=120) and nr_running == 1 The task remains stuck in the wrong LLC indefinitely.
Fix this by exempting migrate_llc_task from the sched_asym filter.
--- kernel/sched/fair.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
kernel/sched/fair.c +5 -1
@@ -12986,8 +12986,12 @@ static struct rq *sched_balance_find_src_rq(struct lb_env *env,* * If balancing between cores, let lower priority CPUs help * SMT cores with more than one busy sibling. +* +* For migrate_llc_task, skip this check: cache locality +* outweighs the asym priority. */ -if (sched_asym (env->sd, i, env->dst_cpu) && nr_running ==1 ) +if (sched_asym (env->sd, i, env->dst_cpu) && nr_running ==1 && + env->migration_type != migrate_llc_task)continue ;switch (env->migration_type) {
Signature · 1 line
2.25.1
kernel/sched/fair.c static struct rq *sched_balance_find_src_rq(struct lb_env *env, · line 12995
Quoted 4 lines · Chen Yu
kernel/sched/fair.c +2
-1-if (sched_asym (env->sd, i, env->dst_cpu) && nr_running ==1 ) +if (sched_asym (env->sd, i, env->dst_cpu) && nr_running ==1 && + env->migration_type != migrate_llc_task)continue ;
The fix looks correct for the ITMT case. I also convinced myself it does not introduce a migrate_llc_task <-> asym-packing ping-pong: once the task lands in its preferred LLC, a subsequent asym pull-back is a migrate_task, which goes through can_migrate_task() -> migrate_degrades_llc() -> can_migrate_llc_task() == mig_forbid (moving away from the preferred LLC), so it returns 0 and the task is not pulled back (modulo the nr_balance_failed >= cache_nice_tries + 1 escape). So it should not oscillate. Nice.
One minor consistency question: the adjacent single-task filter just above,
if (env->sd->flags & SD_ASYM_CPUCAPACITY && !capacity_greater(capacity_of(env->dst_cpu), capacity) && nr_running == 1) continue;
has the same nr_running == 1 shape and is not exempted for migrate_llc_task. I don't think it is reachable on current hardware (it would need both multiple LLCs, so migrate_llc_task can fire, and asymmetric CPU capacity at that domain), so this is not a correctness concern for the reported case. But since the two filters now treat migrate_llc_task differently, was leaving the capacity one intentional (capacity prioritized over cache locality on asym-capacity), or just out of scope here? A word in the changelog would make the intent clear.
Thanks,
Xusheng
Hi Xusheng,
Quoted 15 lines · Zhan Xusheng
kernel/sched/fair.c static struct rq *sched_balance_find_src_rq(struct lb_env *env, · line 12995
Quoted 4 lines · Chen Yu
kernel/sched/fair.c +2
-1-if (sched_asym (env->sd, i, env->dst_cpu) && nr_running ==1 ) +if (sched_asym (env->sd, i, env->dst_cpu) && nr_running ==1 && + env->migration_type != migrate_llc_task)continue ;The fix looks correct for the ITMT case. I also convinced myself it does not introduce a
migrate_llc_task<-> asym-packing ping-pong: once the task lands in its preferred LLC, a subsequent asym pull-back is amigrate_task, which goes throughcan_migrate_task()->migrate_degrades_llc()->can_migrate_llc_task()==mig_forbid(moving away from the preferred LLC), so it returns 0 and the task is not pulled back (modulo thenr_balance_failed>=cache_nice_tries+ 1 escape). So it should not oscillate. Nice.
Thanks for checking the ping-pong angle, and I agree it does not oscillate. I did a further investigation, and another reason it holds: with only one running task, migrate_degrades_llc() is not what keeps the task in its preferred LLC. In the regular load balance path can_migrate_task() already bails at task_on_cpu() - since that task is curr. And in the active balance path alb_break_llc() blocks it first:
if (env->src_rq->nr_running <= 1)
return true;
so the task is not pulled back either way.
Quoted 7 lines
One minor consistency question: the adjacent single-task filter just above,
if (env->sd->flags & SD_ASYM_CPUCAPACITY && !capacity_greater(capacity_of(env->dst_cpu), capacity) && nr_running == 1) continue;
Quoted 9 lines
has the same
nr_running== 1 shape and is not exempted formigrate_llc_task. I don't think it is reachable on current hardware (it would need both multiple LLCs, somigrate_llc_taskcan fire, and asymmetric CPU capacity at that domain), so this is not a correctness concern for the reported case. But since the two filters now treatmigrate_llc_taskdifferently, was leaving the capacity one intentional (capacity prioritized over cache locality on asym-capacity), or just out of scope here? A word in the changelog would make the intent clear.
Good point. OK, let me add description in changelog:
The adjacent SD_ASYM_CPUCAPACITY filter was left unchanged. In theory it should have been handled in the same way, but it is not clear yet whether any platform has both multiple LLCs and asymmetric CPU capacity, so the asymmetric CPU capacity logic was not fine-tuned here and can be revisited once such a platform shows up. In theory the priority follows the order of enum group_type: the priority of group_llc_balance is lower than that of group_overloaded and higher than that of all other group types.
thanks,
Chenyu
Zhan Xusheng
Reviewed-byFrom: Zhan Xusheng <[email protected]>
Quoted 10 lines · Yu C
I did a further investigation, and another reason it holds: with only one running task,
migrate_degrades_llc()is not what keeps the task in its preferred LLC. In the regular load balance pathcan_migrate_task()already bails attask_on_cpu()- since that task is curr. And in the active balance pathalb_break_llc()blocks it first:if (
env->src_rq->nr_running<= 1)return true;so the task is not pulled back either way.
Thanks, that all makes sense. I traced both guards and they hold:
- Regular load balance: for the single running task can_migrate_task() returns 0 at the task_on_cpu() check, so it is never pulled back on the normal path (and for a non-running task the migrate_degrades_llc() -> can_migrate_llc_task() == mig_forbid check catches it).
- Active balance: need_active_balance() -> alb_break_llc() returns true because nr_pref_llc_running == h_nr_runnable and src_rq->nr_running <= 1. What convinced me it is not symmetric - i.e. that the forward migrate_llc_task still works - is that in the forward direction the task does not prefer the source LLC, so nr_pref_llc_running == 0 there and alb_break_llc() returns false.
Quoted 6 lines
Good point. OK, let me add description in changelog:
The adjacentSD_ASYM_CPUCAPACITYfilter was left unchanged. In theory it should have been handled in the same way, but it is not clear yet whether any platform has both multiple LLCs and asymmetric CPU capacity, so the asymmetric CPU capacity logic was not fine-tuned here and can be revisited once such a platform shows up.
That wording answers my question well - thanks for spelling out that leaving the capacity filter alone is intentional pending a multi-LLC + asym-capacity platform.
Thanks,
Xusheng
Tim Chen
Reviewed-byQuoted 35 lines · Zhan Xusheng
From: Zhan Xusheng <[email protected]>
Quoted 10 lines · Yu C
I did a further investigation, and another reason it holds: with only one running task,
migrate_degrades_llc()is not what keeps the task in its preferred LLC. In the regular load balance pathcan_migrate_task()already bails attask_on_cpu()- since that task is curr. And in the active balance pathalb_break_llc()blocks it first:if (
env->src_rq->nr_running<= 1)return true;so the task is not pulled back either way.
Thanks, that all makes sense. I traced both guards and they hold:
- Regular load balance: for the single running task
can_migrate_task()returns 0 at thetask_on_cpu()check, so it is never pulled back on the normal path (and for a non-running task themigrate_degrades_llc()->can_migrate_llc_task()==mig_forbidcheck catches it).- Active balance:
need_active_balance()->alb_break_llc()returns true becausenr_pref_llc_running==h_nr_runnableandsrc_rq->nr_running<= 1. What convinced me it is not symmetric - i.e. that the forwardmigrate_llc_taskstill works - is that in the forward direction the task does not prefer the source LLC, sonr_pref_llc_running== 0 there andalb_break_llc()returns false.Quoted 6 lines
Good point. OK, let me add description in changelog:
The adjacentSD_ASYM_CPUCAPACITYfilter was left unchanged. In theory it should have been handled in the same way, but it is not clear yet whether any platform has both multiple LLCs and asymmetric CPU capacity, so the asymmetric CPU capacity logic was not fine-tuned here and can be revisited once such a platform shows up.
Quoted 3 lines
That wording answers my question well - thanks for spelling out that leaving the capacity filter alone is intentional pending a multi-LLC + asym-capacity platform.
My understanding is that SD_ASYM_CPUCAPACITY is mostly used in the hybrid CPU situations (e.g. P-core and E-core on some Intel client CPUs). For those, the difference in CPU performance is pretty significant and we likely will not get as much performance back from cache co-location vs the CPU capacity. So I think we can leave the SD_ASYM_CPUCAPACITY code as is till we come across a workload and platform that shows otherwise.
Otherwise,
Tim
Quoted 5 lines
Reviewed-by Zhan Xusheng <[email protected]>Thanks,
Xusheng