Skip to contentSkip to conversation

sched/cache: Fix a thread aggregation conflict when there is one runnable task

PATCH Superseded Chen Yu 3 participants 5 messages Jul 27 – Jul 28 Scheduler

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore

Chen Yu

author
#0 lore raw

PATCH 1/1 sched/cache: Fix a thread aggregation conflict when there is one runnable task

On systems with SD_ASYM_PACKING set at the PKG domain level (e.g., with ITMT enabled), the sched_asym() check in sched_balance_find_src_rq() prevents pulling a single task from a higher-priority CPU to a lower-priority CPU. This blocks migrate_llc_task migrations where a task wants to move to its preferred LLC for cache locality.

For example, CPU 0 (highest priority) has a task preferring LLC3, but the load balancer on CPU 120 (LLC3) cannot pull it because:
sched_asym(PKG_sd, cpu=0, dst_cpu=120) and nr_running == 1 The task remains stuck in the wrong LLC indefinitely.

Fix this by exempting migrate_llc_task from the sched_asym filter.

Fixes e4c9a4cb244a ("sched/cache: Add migrate_llc_task migration type for cache-aware balancing")
---
 kernel/sched/fair.c | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
kernel/sched/fair.c +5 -1
index d78467ec6ee1..e6e6fa84c9e1 100644
@@ -12986,8 +12986,12 @@ static struct rq *sched_balance_find_src_rq(struct lb_env *env,
 		 *
 		 * If balancing between cores, let lower priority CPUs help
 		 * SMT cores with more than one busy sibling.
+		 *
+		 * For migrate_llc_task, skip this check: cache locality
+		 * outweighs the asym priority.
 		 */
-		if (sched_asym(env->sd, i, env->dst_cpu) && nr_running == 1)
+		if (sched_asym(env->sd, i, env->dst_cpu) && nr_running == 1 &&
+		    env->migration_type != migrate_llc_task)
 			continue;
 
 		switch (env->migration_type) {
Signature · 1 line
2.25.1
#1 lore raw

kernel/sched/fair.c static struct rq *sched_balance_find_src_rq(struct lb_env *env, · line 12995

Quoted 4 lines · Chen Yu
kernel/sched/fair.c +2 -1
-	if (sched_asym(env->sd, i, env->dst_cpu) && nr_running == 1)
+	if (sched_asym(env->sd, i, env->dst_cpu) && nr_running == 1 &&
+	    env->migration_type != migrate_llc_task)
 		continue;

The fix looks correct for the ITMT case. I also convinced myself it does not introduce a migrate_llc_task <-> asym-packing ping-pong: once the task lands in its preferred LLC, a subsequent asym pull-back is a migrate_task, which goes through can_migrate_task() -> migrate_degrades_llc() -> can_migrate_llc_task() == mig_forbid (moving away from the preferred LLC), so it returns 0 and the task is not pulled back (modulo the nr_balance_failed >= cache_nice_tries + 1 escape). So it should not oscillate. Nice.

One minor consistency question: the adjacent single-task filter just above,

	if (env->sd->flags & SD_ASYM_CPUCAPACITY &&
	    !capacity_greater(capacity_of(env->dst_cpu), capacity) &&
	    nr_running == 1)
		continue;

has the same nr_running == 1 shape and is not exempted for migrate_llc_task. I don't think it is reachable on current hardware (it would need both multiple LLCs, so migrate_llc_task can fire, and asymmetric CPU capacity at that domain), so this is not a correctness concern for the reported case. But since the two filters now treat migrate_llc_task differently, was leaving the capacity one intentional (capacity prioritized over cache locality on asym-capacity), or just out of scope here? A word in the changelog would make the intent clear.

Thanks,
Xusheng

author
#2 lore raw

Hi Xusheng,

Quoted 15 lines · Zhan Xusheng

kernel/sched/fair.c static struct rq *sched_balance_find_src_rq(struct lb_env *env, · line 12995

Quoted 4 lines · Chen Yu
kernel/sched/fair.c +2 -1
-	if (sched_asym(env->sd, i, env->dst_cpu) && nr_running == 1)
+	if (sched_asym(env->sd, i, env->dst_cpu) && nr_running == 1 &&
+	    env->migration_type != migrate_llc_task)
  		continue;

The fix looks correct for the ITMT case. I also convinced myself it does not introduce a migrate_llc_task <-> asym-packing ping-pong: once the task lands in its preferred LLC, a subsequent asym pull-back is a migrate_task, which goes through can_migrate_task() -> migrate_degrades_llc() -> can_migrate_llc_task() == mig_forbid (moving away from the preferred LLC), so it returns 0 and the task is not pulled back (modulo the nr_balance_failed >= cache_nice_tries + 1 escape). So it should not oscillate. Nice.

Thanks for checking the ping-pong angle, and I agree it does not oscillate. I did a further investigation, and another reason it holds: with only one running task, migrate_degrades_llc() is not what keeps the task in its preferred LLC. In the regular load balance path can_migrate_task() already bails at task_on_cpu() - since that task is curr. And in the active balance path alb_break_llc() blocks it first:

if (env->src_rq->nr_running <= 1)

    return true;

so the task is not pulled back either way.

Quoted 7 lines

One minor consistency question: the adjacent single-task filter just above,

	if (env->sd->flags & SD_ASYM_CPUCAPACITY &&
	    !capacity_greater(capacity_of(env->dst_cpu), capacity) &&
	    nr_running == 1)
		continue;
Quoted 9 lines

has the same nr_running == 1 shape and is not exempted for migrate_llc_task. I don't think it is reachable on current hardware (it would need both multiple LLCs, so migrate_llc_task can fire, and asymmetric CPU capacity at that domain), so this is not a correctness concern for the reported case. But since the two filters now treat migrate_llc_task differently, was leaving the capacity one intentional (capacity prioritized over cache locality on asym-capacity), or just out of scope here? A word in the changelog would make the intent clear.

Good point. OK, let me add description in changelog:
The adjacent SD_ASYM_CPUCAPACITY filter was left unchanged. In theory it should have been handled in the same way, but it is not clear yet whether any platform has both multiple LLCs and asymmetric CPU capacity, so the asymmetric CPU capacity logic was not fine-tuned here and can be revisited once such a platform shows up. In theory the priority follows the order of enum group_type: the priority of group_llc_balance is lower than that of group_overloaded and higher than that of all other group types.

thanks,
Chenyu

Reviewed-by
#3 lore raw

From: Zhan Xusheng <[email protected]>

Quoted 10 lines · Yu C

I did a further investigation, and another reason it holds: with only one running task, migrate_degrades_llc() is not what keeps the task in its preferred LLC. In the regular load balance path can_migrate_task() already bails at task_on_cpu() - since that task is curr. And in the active balance path alb_break_llc() blocks it first:

if (env->src_rq->nr_running <= 1)

     return true;

so the task is not pulled back either way.

Thanks, that all makes sense. I traced both guards and they hold:

- Regular load balance: for the single running task can_migrate_task() returns 0 at the task_on_cpu() check, so it is never pulled back on the normal path (and for a non-running task the migrate_degrades_llc() -> can_migrate_llc_task() == mig_forbid check catches it).

- Active balance: need_active_balance() -> alb_break_llc() returns true because nr_pref_llc_running == h_nr_runnable and src_rq->nr_running <= 1. What convinced me it is not symmetric - i.e. that the forward migrate_llc_task still works - is that in the forward direction the task does not prefer the source LLC, so nr_pref_llc_running == 0 there and alb_break_llc() returns false.

Quoted 6 lines

Good point. OK, let me add description in changelog:
The adjacent SD_ASYM_CPUCAPACITY filter was left unchanged. In theory it should have been handled in the same way, but it is not clear yet whether any platform has both multiple LLCs and asymmetric CPU capacity, so the asymmetric CPU capacity logic was not fine-tuned here and can be revisited once such a platform shows up.

That wording answers my question well - thanks for spelling out that leaving the capacity filter alone is intentional pending a multi-LLC + asym-capacity platform.

Thanks,
Xusheng

Reviewed-by
#4 lore raw
Quoted 35 lines · Zhan Xusheng

From: Zhan Xusheng <[email protected]>

Quoted 10 lines · Yu C

I did a further investigation, and another reason it holds: with only one running task, migrate_degrades_llc() is not what keeps the task in its preferred LLC. In the regular load balance path can_migrate_task() already bails at task_on_cpu() - since that task is curr. And in the active balance path alb_break_llc() blocks it first:

if (env->src_rq->nr_running <= 1)

     return true;

so the task is not pulled back either way.

Thanks, that all makes sense. I traced both guards and they hold:

- Regular load balance: for the single running task can_migrate_task() returns 0 at the task_on_cpu() check, so it is never pulled back on the normal path (and for a non-running task the migrate_degrades_llc() -> can_migrate_llc_task() == mig_forbid check catches it).

- Active balance: need_active_balance() -> alb_break_llc() returns true because nr_pref_llc_running == h_nr_runnable and src_rq->nr_running <= 1. What convinced me it is not symmetric - i.e. that the forward migrate_llc_task still works - is that in the forward direction the task does not prefer the source LLC, so nr_pref_llc_running == 0 there and alb_break_llc() returns false.

Quoted 6 lines

Good point. OK, let me add description in changelog:
The adjacent SD_ASYM_CPUCAPACITY filter was left unchanged. In theory it should have been handled in the same way, but it is not clear yet whether any platform has both multiple LLCs and asymmetric CPU capacity, so the asymmetric CPU capacity logic was not fine-tuned here and can be revisited once such a platform shows up.

Quoted 3 lines

That wording answers my question well - thanks for spelling out that leaving the capacity filter alone is intentional pending a multi-LLC + asym-capacity platform.

My understanding is that SD_ASYM_CPUCAPACITY is mostly used in the hybrid CPU situations (e.g. P-core and E-core on some Intel client CPUs). For those, the difference in CPU performance is pretty significant and we likely will not get as much performance back from cache co-location vs the CPU capacity. So I think we can leave the SD_ASYM_CPUCAPACITY code as is till we come across a workload and platform that shows otherwise.

Otherwise,

Tim

Quoted 5 lines

Thanks,
Xusheng