sched/fair: SMT-aware asymmetric CPU capacity
v2 cover sched/fair: SMT-aware asymmetric CPU capacity
This series attempts to improve SD_ASYM_CPUCAPACITY scheduling by introducing SMT awareness.
= Problem =
Nominal per-logical-CPU capacity can overstate usable compute when an SMT sibling is busy, because the physical core doesn't deliver its full nominal capacity. So, several asym-cpu-capacity paths may pick high capacity idle CPUs that are not actually good destinations.
= Solution =
This patch set aligns those paths with a simple rule already used elsewhere:
when SMT is active, prefer fully idle cores and avoid treating partially idle SMT siblings as full-capacity targets where that would mislead load balance.
Patch set summary:
- Prefer fully-idle SMT cores in asym-capacity idle selection: in the wakeup fast path, extend select_idle_capacity() / asym_fits_cpu() so idle selection can prefer CPUs on fully idle cores.
- Reject misfit pulls onto busy SMT siblings on SD_ASYM_CPUCAPACITY.
This patch set has been tested on the new Vera Rubin platform, where SMT is enabled and the firmware exposes small frequency variations (+/-~5%) as differences in CPU capacity, resulting in SD_ASYM_CPUCAPACITY being set.
Without these patches, performance can drop by up to ~2x with CPU-intensive workloads, because the SD_ASYM_CPUCAPACITY idle selection policy does not account for busy SMT siblings.
Alternative approaches have been evaluated, such as equalizing CPU capacities, either by exposing uniform values via firmware or normalizing them in the kernel by grouping CPUs within a small capacity window (+-5%).
However, the SMT-aware SD_ASYM_CPUCAPACITY approach has shown better results so far. Improving this policy also seems worthwhile in general, as future platforms may enable SMT with asymmetric CPU topologies.
Performance results on Vera Rubin with SD_ASYM_CPUCAPACITY (mainline) vs SD_ASYM_CPUCAPACITY + SMT:
- NVBLAS benchblas (one task / SMT core):
+---------------------------------+--------+ | Configuration | gflops | +---------------------------------+--------+ | ASYM (mainline) + SIS_UTIL | 5478 | | ASYM (mainline) + NO_SIS_UTIL | 5491 | | | | | NO ASYM + SIS_UTIL | 8912 | | NO ASYM + NO_SIS_UTIL | 8978 | | | | | ASYM + SMT + SIS_UTIL | 9259 | | ASYM + SMT + NO_SIS_UTIL | 9291 | +---------------------------------+--------+
- DCPerf MediaWiki (all CPUs):
+---------------------------------+--------+--------+--------+--------+ | Configuration | rps | p50 | p95 | p99 | +---------------------------------+--------+--------+--------+--------+ | ASYM (mainline) + SIS_UTIL | 7994 | 0.052 | 0.223 | 0.246 | | ASYM (mainline) + NO_SIS_UTIL | 7993 | 0.052 | 0.221 | 0.245 | | | | | | | | NO ASYM + SIS_UTIL | 8113 | 0.067 | 0.184 | 0.225 | | NO ASYM + NO_SIS_UTIL | 8093 | 0.068 | 0.184 | 0.223 | | | | | | | | ASYM + SMT + SIS_UTIL | 8129 | 0.076 | 0.149 | 0.188 | | ASYM + SMT + NO_SIS_UTIL | 8138 | 0.076 | 0.148 | 0.186 | +---------------------------------+--------+--------+--------+--------+
In the MediaWiki case SMT awareness is less impactful (compared to equalizing CPU capacities), because for the majority of the run all CPUs are used, but it still seems to provide some benefits at reducing tail latency.
See also:
- https://lore.kernel.org/lkml/[email protected]
- https://lore.kernel.org/lkml/[email protected]
Changes in v2:
- Rework SMT awareness logic in select_idle_capacity() (K Prateek Nayak)
- Drop EAS and find_new_ilb() changes for now
- Link to v1: https://lore.kernel.org/all/[email protected]
Andrea Righi (2):
sched/fair: Prefer fully-idle SMT cores in asym-capacity idle selection sched/fair: Reject misfit pulls onto busy SMT siblings on asym-capacity
kernel/sched/fair.c | 44 +++++++++++++++++++++++++++++++++++++++----- 1 file changed, 39 insertions(+), 5 deletions(-)
PATCH v2 1/2 sched/fair: Prefer fully-idle SMT cores in asym-capacity idle selection
On systems with asymmetric CPU capacity (e.g., ACPI/CPPC reporting different per-core frequencies), the wakeup path uses select_idle_capacity() and prioritizes idle CPUs with higher capacity for better task placement.
However, when those CPUs belong to SMT cores, their effective capacity can be much lower than the nominal capacity when the sibling thread is
busy: SMT siblings compete for shared resources, so a "high capacity" CPU that is idle but whose sibling is busy does not deliver its full capacity. This effective capacity reduction cannot be modeled by the static capacity value alone.
When SMT is active, teach asym-capacity idle selection to treat a logical CPU as a weaker target if its physical core is only partially
idle: select_idle_capacity() no longer returns on the first idle CPU whose static capacity fits the task when that CPU still has a busy sibling, it keeps scanning for an idle CPU on a fully-idle core and only if none qualify does it fall back to partially-idle cores, using shifted fit scores so fully-idle cores win ties; asym_fits_cpu() applies the same fully-idle core requirement when asym capacity and SMT are both active.
This improves task placement, since partially-idle SMT siblings deliver less than their nominal capacity. Favoring fully idle cores, when available, can significantly enhance both throughput and wakeup latency on systems with both SMT and CPU asymmetry.
No functional changes on systems with only asymmetric CPUs or only SMT.
--- kernel/sched/fair.c | 36 ++++++++++++++++++++++++++++++++---- 1 file changed, 32 insertions(+), 4 deletions(-)
kernel/sched/fair.c +32 -4
@@ -7774,6 +7774,7 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, boolstatic int select_idle_capacity (struct task_struct *p,struct sched_domain *sd,int target) { +bool prefers_idle_core =sched_smt_active () &&test_idle_cores (target);unsigned long task_util, util_min, util_max, best_cap =0 ;int fits, best_fits =0 ;int cpu, best_cpu =-1 ;
@@ -7787,6 +7788,7 @@ select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int target) util_max =uclamp_eff_value (p,UCLAMP_MAX );for_each_cpu_wrap (cpu, cpus, target) { +bool preferred_core = !prefers_idle_core ||is_core_idle (cpu);unsigned long cpu_cap =capacity_of (cpu);if (!available_idle_cpu (cpu) && !sched_idle_cpu (cpu))
@@ -7795,7 +7797,7 @@ select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int target) fits =util_fits_cpu (task_util, util_min, util_max, cpu);/* This CPU fits with all requirements */ -if (fits >0 ) +if (fits >0 && preferred_core)return cpu;/* * Only the min performance hint (i.e. uclamp_min) doesn't fit.
@@ -7803,9 +7805,30 @@ select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int target)*/ else if (fits < 0) cpu_cap =get_actual_cpu_capacity (cpu); +/* +* fits > 0 implies we are not on a preferred core +* but the util fits CPU capacity. Set fits to -2 so +* the effective range becomes [-2, 0] where: +* 0 - does not fit +* -1 - fits with the exception of UCLAMP_MIN +* -2 - fits with the exception of preferred_core +*/ + else if (fits > 0) + fits =-2 ; + +/* +* If we are on a preferred core, translate the range of fits +* of [-1, 0] to [-4, -3]. This ensures that an idle core +* is always given priority over (partially) busy core. +* +* A fully fitting idle core would have returned early and hence +* fits > 0 for preferred_core need not be dealt with. +*/ +if (preferred_core) + fits -=3 ;/* -* First, select CPU which fits better (-1 being better than 0). +* First, select CPU which fits better (lower is more preferred). * Then, select the one with best capacity at same level. */ if ((fits < best_fits) ||
@@ -7824,12 +7847,17 @@ static inline bool asym_fits_cpu(unsigned long util, unsigned long util_max, int cpu) { - if (sched_asym_cpucap_active()) + if (sched_asym_cpucap_active()) {/* * Return true only if the cpu fully fits the task requirements * which include the utilization and the performance hints. +* +* When SMT is active, also require that the core has no busy +* siblings. */ - return (util_fits_cpu(util, util_min, util_max, cpu) > 0); +return (!sched_smt_active () ||is_core_idle (cpu)) && + (util_fits_cpu (util, util_min, util_max, cpu) >0 ); + }return true; }
Signature · 1 line
2.53.0
Dietmar Eggemann
reviewerRe: [PATCH 1/2] sched/fair: Prefer fully-idle SMT cores in asym-capacity idle selection
Quoted 45 lines · Andrea Righi
On systems with asymmetric CPU capacity (e.g., ACPI/CPPC reporting different per-core frequencies), the wakeup path uses
select_idle_capacity()and prioritizes idle CPUs with higher capacity for better task placement.However, when those CPUs belong to SMT cores, their effective capacity can be much lower than the nominal capacity when the sibling thread is
busy: SMT siblings compete for shared resources, so a "high capacity" CPU that is idle but whose sibling is busy does not deliver its full capacity. This effective capacity reduction cannot be modeled by the static capacity value alone.When SMT is active, teach asym-capacity idle selection to treat a logical CPU as a weaker target if its physical core is only partially
idle:select_idle_capacity()no longer returns on the first idle CPU whose static capacity fits the task when that CPU still has a busy sibling, it keeps scanning for an idle CPU on a fully-idle core and only if none qualify does it fall back to partially-idle cores, using shifted fit scores so fully-idle cores win ties;asym_fits_cpu()applies the same fully-idle core requirement when asym capacity and SMT are both active.This improves task placement, since partially-idle SMT siblings deliver less than their nominal capacity. Favoring fully idle cores, when available, can significantly enhance both throughput and wakeup latency on systems with both SMT and CPU asymmetry.
No functional changes on systems with only asymmetric CPUs or only SMT.
Cc K Prateek Nayak <[email protected]>Cc Vincent Guittot <[email protected]>Cc Dietmar Eggemann <[email protected]>Cc Christian Loehle <[email protected]>Cc Koba Ko <[email protected]>Reported-by Felix Abecassis <[email protected]>Signed-off-by Andrea Righi <[email protected]>--- kernel/sched/fair.c | 36 ++++++++++++++++++++++++++++++++---- 1 file changed, 32 insertions(+), 4 deletions(-)kernel/sched/fair.c +0
-0
kernel/sched/fair.c static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, bool · line 7777
Quoted 4 lines
kernel/sched/fair.c +1
-0@@ -7774,6 +7774,7 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, boolstatic int select_idle_capacity (struct task_struct *p,struct sched_domain *sd,int target) { +bool prefers_idle_core =sched_smt_active () &&test_idle_cores (target);
Somehow I miss a:
if (prefers_idle_core)
set_idle_cores(target, false)
The one in select_idle_sibling() -> select_idle_cpu() isn't executed anymore in with ASYM_CPUCAPACITY.
Another thing is that sic() iterates over CPUs sd_asym_cpucapacity whereas the idle core thing lives in sd_llc/sd_llc_shared. Both sd's are probably th same on your system.
Quoted 3 lines
kernel/sched/fair.c +0
-0unsigned long task_util, util_min, util_max, best_cap =0 ;int fits, best_fits =0 ;int cpu, best_cpu =-1 ;
[...]
Re: [PATCH 1/2] sched/fair: Prefer fully-idle SMT cores in asym-capacity idle selection
Hi Dietmar,
Quoted 55 lines · Dietmar Eggemann
Quoted 45 lines · Andrea Righi
On systems with asymmetric CPU capacity (e.g., ACPI/CPPC reporting different per-core frequencies), the wakeup path uses
select_idle_capacity()and prioritizes idle CPUs with higher capacity for better task placement.However, when those CPUs belong to SMT cores, their effective capacity can be much lower than the nominal capacity when the sibling thread is
busy: SMT siblings compete for shared resources, so a "high capacity" CPU that is idle but whose sibling is busy does not deliver its full capacity. This effective capacity reduction cannot be modeled by the static capacity value alone.When SMT is active, teach asym-capacity idle selection to treat a logical CPU as a weaker target if its physical core is only partially
idle:select_idle_capacity()no longer returns on the first idle CPU whose static capacity fits the task when that CPU still has a busy sibling, it keeps scanning for an idle CPU on a fully-idle core and only if none qualify does it fall back to partially-idle cores, using shifted fit scores so fully-idle cores win ties;asym_fits_cpu()applies the same fully-idle core requirement when asym capacity and SMT are both active.This improves task placement, since partially-idle SMT siblings deliver less than their nominal capacity. Favoring fully idle cores, when available, can significantly enhance both throughput and wakeup latency on systems with both SMT and CPU asymmetry.
No functional changes on systems with only asymmetric CPUs or only SMT.
Cc K Prateek Nayak <[email protected]>Cc Vincent Guittot <[email protected]>Cc Dietmar Eggemann <[email protected]>Cc Christian Loehle <[email protected]>Cc Koba Ko <[email protected]>Reported-by Felix Abecassis <[email protected]>Signed-off-by Andrea Righi <[email protected]>--- kernel/sched/fair.c | 36 ++++++++++++++++++++++++++++++++---- 1 file changed, 32 insertions(+), 4 deletions(-)kernel/sched/fair.c +0
-0kernel/sched/fair.c static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, bool · line 7777
Quoted 4 lines
kernel/sched/fair.c +1
-0@@ -7774,6 +7774,7 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, boolstatic int select_idle_capacity (struct task_struct *p,struct sched_domain *sd,int target) { +bool prefers_idle_core =sched_smt_active () &&test_idle_cores (target);Somehow I miss a:
Quoted 6 lines
if (
prefers_idle_core)
set_idle_cores(target, false)The one in
select_idle_sibling()->select_idle_cpu()isn't executed anymore in withASYM_CPUCAPACITY.
Right, we need to add this as also pointed by Vincent.
Quoted 4 lines
Another thing is that
sic()iterates over CPUssd_asym_cpucapacitywhereas the idle core thing lives in sd_llc/sd_llc_shared. Both sd's are probably th same on your system.
Hm... they're the same on my machine, but if they're different, clearing has_idle_cores here is not right and it might lead to false positives. We should only clear it only when both domains span the same CPUs (or just check if sd_asym_cpucapacity and sd_llc are the same).
However, if they're not the same, I'm not sure exactly what we should do... maybe ignore has_idle_cores and always do the scan for now?
Thanks,
-Andrea
K Prateek Nayak
reviewerRe: [PATCH 1/2] sched/fair: Prefer fully-idle SMT cores in asym-capacity idle selection
Hello Andrea,
Quoted 22 lines · Andrea Righi
Quoted 8 lines
Review static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, bool · line 7777
Quoted 5 lines
(quoted diff) +1
-0@@ -7774,6 +7774,7 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, bool static int select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int target) { + bool prefers_idle_core = sched_smt_active() && test_idle_cores(target);Somehow I miss a:
Quoted 6 lines
if (
prefers_idle_core)
set_idle_cores(target, false)The one in
select_idle_sibling()->select_idle_cpu()isn't executed anymore in withASYM_CPUCAPACITY.Right, we need to add this as also pointed by Vincent.
Quoted 4 lines
Another thing is that
sic()iterates over CPUssd_asym_cpucapacitywhereas the idle core thing lives in sd_llc/sd_llc_shared. Both sd's are probably th same on your system.
Quoted 7 lines
Hm... they're the same on my machine, but if they're different, clearing
has_idle_coreshere is not right and it might lead to false positives. We should only clear it only when both domains span the same CPUs (or just check ifsd_asym_cpucapacityandsd_llcare the same).However, if they're not the same, I'm not sure exactly what we should do... maybe ignore
has_idle_coresand always do the scan for now?
With your changes, only two places actually care about test_idle_cores():
- select_idle_capacity()
- select_idle_cpu()
If we go into select_idle_capacity(), we don't do select_idle_cpu() so the two paths are mutually exclusive.
In nohz_balancer_kick(), if we find, sd_asym_cpucapacity, we simply don't care about the sd_llc_shared->nr_busy_cpus during balancing so that begs the question if we can simply track idle_cores at sd_asym_cpucapacity for these systems?
Following is only build tested for now but I'll try to spoof asym cpucapacity on my system and check if it holds up or not:
(On top of tip:sched/core at sched-core-2026-04-13 + this series)
kernel/sched/fair.c +5 -3
@@ -7913,7 +7913,7 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, boolstruct cpumask *cpus =this_cpu_cpumask_var_ptr (select_rq_mask);int i, cpu, idle_cpu =-1 , nr =INT_MAX ; -if (sched_feat (SIS_UTIL )) { +if (sched_feat (SIS_UTIL ) && sd->shared) {/* * Increment because !--nr is the condition to stop scan. *
@@ -12856,7 +12856,8 @@ static void set_cpu_sd_state_busy(int cpu) goto unlock; sd->nohz_idle =0 ; -atomic_inc (&sd->shared->nr_busy_cpus); +if (sd->shared) +atomic_inc (&sd->shared->nr_busy_cpus); unlock:rcu_read_unlock (); }
@@ -12885,7 +12886,8 @@ static void set_cpu_sd_state_idle(int cpu) goto unlock; sd->nohz_idle =1 ; -atomic_dec (&sd->shared->nr_busy_cpus); +if (sd->shared) +atomic_dec (&sd->shared->nr_busy_cpus); unlock:rcu_read_unlock (); }
kernel/sched/topology.c +53 -11
103 lines not on this page · raw message
---
I still have one question: Can first SD_ASYM_CPUCAPACITY_FULL be set at a SD_NUMA?
We'll need to deal with overlapping domains then but seems like it could be possible with weird cpusets :-(
But in that case, do we even want to search CPUs outside the NUMA in select_idle_capacity()? I don't think anything stops this currently but I might be wrong.
Signature · 2 lines
Thanks and Regards, Prateek
Re: [PATCH 1/2] sched/fair: Prefer fully-idle SMT cores in asym-capacity idle selection
Hi Prateek,
Quoted 38 lines · K Prateek Nayak
Hello Andrea,
Quoted 22 lines · Andrea Righi
Quoted 8 lines
Review static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, bool · line 7777
Quoted 5 lines
(quoted diff) +1
-0@@ -7774,6 +7774,7 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, bool static int select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int target) { + bool prefers_idle_core = sched_smt_active() && test_idle_cores(target);Somehow I miss a:
Quoted 6 lines
if (
prefers_idle_core)
set_idle_cores(target, false)The one in
select_idle_sibling()->select_idle_cpu()isn't executed anymore in withASYM_CPUCAPACITY.Right, we need to add this as also pointed by Vincent.
Quoted 4 lines
Another thing is that
sic()iterates over CPUssd_asym_cpucapacitywhereas the idle core thing lives in sd_llc/sd_llc_shared. Both sd's are probably th same on your system.Quoted 7 lines
Hm... they're the same on my machine, but if they're different, clearing
has_idle_coreshere is not right and it might lead to false positives. We should only clear it only when both domains span the same CPUs (or just check ifsd_asym_cpucapacityandsd_llcare the same).However, if they're not the same, I'm not sure exactly what we should do... maybe ignore
has_idle_coresand always do the scan for now?With your changes, only two places actually care about
test_idle_cores():-
select_idle_capacity()
-select_idle_cpu()
Quoted 7 lines
If we go into
select_idle_capacity(), we don't doselect_idle_cpu()so the two paths are mutually exclusive.In
nohz_balancer_kick(), if we find,sd_asym_cpucapacity, we simply don't care about thesd_llc_shared->nr_busy_cpusduring balancing so that begs the question if we can simply trackidle_coresatsd_asym_cpucapacityfor these systems?
Yeah, makes sense to me. I was planning to test something similar, so thanks for sharing this patch. :) I'll give it a try and report back.
Quoted 155 lines
Following is only build tested for now but I'll try to spoof asym cpucapacity on my system and check if it holds up or not:
(On top of tip:sched/core at sched-core-2026-04-13 + this series)
kernel/sched/fair.c +5
-3@@ -7913,7 +7913,7 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, boolstruct cpumask *cpus =this_cpu_cpumask_var_ptr (select_rq_mask);int i, cpu, idle_cpu =-1 , nr =INT_MAX ; -if (sched_feat (SIS_UTIL )) { +if (sched_feat (SIS_UTIL ) && sd->shared) {/* * Increment because !--nr is the condition to stop scan. * @@ -12856,7 +12856,8 @@ static void set_cpu_sd_state_busy(int cpu) goto unlock; sd->nohz_idle =0 ; -atomic_inc (&sd->shared->nr_busy_cpus); +if (sd->shared) +atomic_inc (&sd->shared->nr_busy_cpus); unlock:rcu_read_unlock (); }@@ -12885,7 +12886,8 @@ static void set_cpu_sd_state_idle(int cpu) goto unlock; sd->nohz_idle =1 ; -atomic_dec (&sd->shared->nr_busy_cpus); +if (sd->shared) +atomic_dec (&sd->shared->nr_busy_cpus); unlock:rcu_read_unlock (); }kernel/sched/topology.c +53
-12@@ -680,19 +680,38 @@ static void update_top_cache_domain(int cpu)int id = cpu;int size =1 ; + sd =lowest_flag_domain (cpu,SD_ASYM_CPUCAPACITY_FULL ); +if (sd) { +/* +* If sd_asym_cpucapacity exists, +* the shared object should exist too. +*/ +WARN_ON_ONCE (!sd->shared); + sds = sd->shared; + } + +rcu_assign_pointer (per_cpu (sd_asym_cpucapacity, cpu), sd); + sd =highest_flag_domain (cpu,SD_SHARE_LLC );if (sd) { id =cpumask_first (sched_domain_span (sd)); size =cpumask_weight (sched_domain_span (sd)); -/* If sd_llc exists, sd_llc_shared should exist too. */ -WARN_ON_ONCE (!sd->shared); - sds = sd->shared; +/* +* If sd_asym_cpucapacity doesn't exist, +* sd_llc_shared must have a sd->shared linked. +*/ +if (!sds) { +WARN_ON_ONCE (!sd->shared); + sds = sd->shared; + } }rcu_assign_pointer (per_cpu (sd_llc, cpu), sd);per_cpu (sd_llc_size, cpu) = size;per_cpu (sd_llc_id, cpu) = id; + +/* TODO: Rename sd_llc_shared to fit the new role. */ rcu_assign_pointer (per_cpu (sd_llc_shared, cpu), sds); sd =lowest_flag_domain (cpu,SD_CLUSTER );@@ -711,9 +730,6 @@ static void update_top_cache_domain(int cpu) sd =highest_flag_domain (cpu,SD_ASYM_PACKING );rcu_assign_pointer (per_cpu (sd_asym_packing, cpu), sd); - - sd =lowest_flag_domain (cpu,SD_ASYM_CPUCAPACITY_FULL ); -rcu_assign_pointer (per_cpu (sd_asym_cpucapacity, cpu), sd); }/* @@ -2650,6 +2666,15 @@ static void adjust_numa_imbalance(struct sched_domain *sd_llc) } } +static void init_sched_domain_shared (struct s_data *d,struct sched_domain *sd) +{ +int sd_id =cpumask_first (sched_domain_span (sd)); + + sd->shared = *per_cpu_ptr (d->sds, sd_id); +atomic_set (&sd->shared->nr_busy_cpus, sd->span_weight); +atomic_inc (&sd->shared->ref); +} +/* * Build sched domains for a given set of CPUs and attach the sched domains * to the individual CPUs @@ -2712,16 +2737,33 @@ build_sched_domains(const struct cpumask *cpu_map, struct sched_domain_attr *attif (!sd)continue ; +/* +* In case of ASYM_CPUCAPACITY, attach sd->shared to +* sd_asym_cpucapacity for wakeup stat tracking. +* +* XXX: This assumes SD_ASYM_CPUCAPACITY_FULL domain +* always has more than one group else it is prone to +* degeneration. +*/ +if (has_asym) { +while (sd && !(sd->flags &SD_ASYM_CPUCAPACITY_FULL )) + sd = sd->parent; + +init_sched_domain_shared (&d, sd); + } +/* First, find the topmost SD_SHARE_LLC domain */ + sd = *per_cpu_ptr (d.sd, i);while (sd->parent && (sd->parent->flags &SD_SHARE_LLC )) sd = sd->parent;if (sd->flags &SD_SHARE_LLC ) { -int sd_id =cpumask_first (sched_domain_span (sd)); - - sd->shared = *per_cpu_ptr (d.sds, sd_id); -atomic_set (&sd->shared->nr_busy_cpus, sd->span_weight); -atomic_inc (&sd->shared->ref); +/* +* Initialize the sd->shared for SD_SHARE_LLC if +* SD_ASYM_CPUCAPACITY_FULL hasn't claimed it already. +*/ +if (!has_asym) +init_sched_domain_shared (&d, sd);/* * In presence of higher domains, adjust the --- I still have one question: Can first
SD_ASYM_CPUCAPACITY_FULLbe set at aSD_NUMA?
Quoted 6 lines
We'll need to deal with overlapping domains then but seems like it could be possible with weird cpusets :-(
But in that case, do we even want to search CPUs outside the NUMA in
select_idle_capacity()? I don't think anything stops this currently but I might be wrong.
My $0.02 on this.
In theory it could happen with unusual topologies or constrained cpusets, although it should be quite rare. That said, select_idle_capacity() already operates on the span of sd_asym_cpucapacity, so if that domain crosses NUMA boundaries, we're already scanning across NUMA today. This patch doesn't fundamentally alter this behavior.
If we think cross-NUMA scanning is undesirable, that's probably a more general issue in select_idle_capacity(), rather than something specific to this change and we can address this later.
Thanks,
-Andrea
K Prateek Nayak
reviewerRe: [PATCH 1/2] sched/fair: Prefer fully-idle SMT cores in asym-capacity idle selection
Hello Andrea,
Quoted 8 lines · Andrea Righi
Quoted 8 lines
With your changes, only two places actually care about
test_idle_cores():-
select_idle_capacity()
-select_idle_cpu()If we go into
select_idle_capacity(), we don't doselect_idle_cpu()so the two paths are mutually exclusive.
Quoted 7 lines
Quoted 4 lines
In
nohz_balancer_kick(), if we find,sd_asym_cpucapacity, we simply don't care about thesd_llc_shared->nr_busy_cpusduring balancing so that begs the question if we can simply trackidle_coresatsd_asym_cpucapacityfor these systems?Yeah, makes sense to me. I was planning to test something similar, so thanks for sharing this patch. :) I'll give it a try and report back.
Thank you for taking it for a spin!
[..snip..]
Quoted 10 lines
Quoted 9 lines
I still have one question: Can first
SD_ASYM_CPUCAPACITY_FULLbe set at aSD_NUMA?We'll need to deal with overlapping domains then but seems like it could be possible with weird cpusets :-(
But in that case, do we even want to search CPUs outside the NUMA in
select_idle_capacity()? I don't think anything stops this currently but I might be wrong.
Quoted 7 lines
My $0.02 on this.
In theory it could happen with unusual topologies or constrained cpusets, although it should be quite rare. That said,
select_idle_capacity()already operates on the span ofsd_asym_cpucapacity, so if that domain crosses NUMA boundaries, we're already scanning across NUMA today. This patch doesn't fundamentally alter this behavior.
Ack! I was just thinking loud from the topology standpoint since sd->shared is not designed to handle the overlapping domains like sg->sgc does but we can probably figure some way to make it work.
Using the ring topology example from topology.c:
0 ----- 1
| | | | | |
3 ----- 2
Consider NUMA-1 below gets the SD_ASYM_CPUCAPACITY_FULL flag:
NUMA-2 0-3 0-3 0-3 0-3
groups: {0-1,3},{1-3} {0-2},{0,2-3} {1-3},{0-1,3} {0,2-3},{0-2} NUMA-1 0-1,3 0-2 1-3 0,2-3
groups: {0},{1},{3} {0},{1},{2} {1},{2},{3} {0},{2},{3}NUMA-0 0 1 2 3
The "sd->shared" assignments at NUMA-1 will put first, second, and the last domain in the same "shared" range by today's logic since the first CPU in their span is the same although their spans are slightly different.
The third will be standalone since the first CPU of the domain span will be different.
Quoted 3 lines
If we think cross-NUMA scanning is undesirable, that's probably a more general issue in
select_idle_capacity(), rather than something specific to this change and we can address this later.
Ack! That is a tangential problem but may require some looking at if we decide to extend the sd->shared object to SD_NUMA domains. I guess if anyone is running such setup, this bit will be the least of their worries.
Signature · 2 lines
Thanks and Regards, Prateek
Re: [PATCH 1/2] sched/fair: Prefer fully-idle SMT cores in asym-capacity idle selection
Hi Prateek,
Quoted 14 lines · K Prateek Nayak
Quoted 13 lines · Andrea Righi
Quoted 12 lines
With your changes, only two places actually care about
test_idle_cores():-
select_idle_capacity()
-select_idle_cpu()If we go into
select_idle_capacity(), we don't doselect_idle_cpu()so the two paths are mutually exclusive.In
nohz_balancer_kick(), if we find,sd_asym_cpucapacity, we simply don't care about thesd_llc_shared->nr_busy_cpusduring balancing so that begs the question if we can simply trackidle_coresatsd_asym_cpucapacityfor these systems?
Quoted 4 lines
Quoted 2 lines
Yeah, makes sense to me. I was planning to test something similar, so thanks for sharing this patch. :) I'll give it a try and report back.
Thank you for taking it for a spin!
I've tested this extensively on Vera and haven't encountered any issues. Performance wise I get similar results (with vs without), which was expected, as sd_llc matches sd_asym_cpucapacity in my case.
Quoted 42 lines
Quoted 11 lines
Quoted 10 lines
I still have one question: Can first
SD_ASYM_CPUCAPACITY_FULLbe set at aSD_NUMA?We'll need to deal with overlapping domains then but seems like it could be possible with weird cpusets :-(
But in that case, do we even want to search CPUs outside the NUMA in
select_idle_capacity()? I don't think anything stops this currently but I might be wrong.Quoted 7 lines
My $0.02 on this.
In theory it could happen with unusual topologies or constrained cpusets, although it should be quite rare. That said,
select_idle_capacity()already operates on the span ofsd_asym_cpucapacity, so if that domain crosses NUMA boundaries, we're already scanning across NUMA today. This patch doesn't fundamentally alter this behavior.Ack! I was just thinking loud from the topology standpoint since
sd->sharedis not designed to handle the overlapping domains likesg->sgcdoes but we can probably figure some way to make it work.Using the ring topology example from
topology.c:0 ----- 1
| | | | | |3 ----- 2
Consider NUMA-1 below gets the
SD_ASYM_CPUCAPACITY_FULLflag:NUMA-2 0-3 0-3 0-3 0-3 groups: {0-1,3},{1-3} {0-2},{0,2-3} {1-3},{0-1,3} {0,2-3},{0-2}NUMA-1 0-1,3 0-2 1-3 0,2-3 groups: {0},{1},{3} {0},{1},{2} {1},{2},{3} {0},{2},{3}NUMA-0 0 1 2 3
Quoted 7 lines
The "
sd->shared" assignments at NUMA-1 will put first, second, and the last domain in the same "shared" range by today's logic since the first CPU in their span is the same although their spans are slightly different.The third will be standalone since the first CPU of the domain span will be different.
Yeah, makes sense. I'm wondering if we should attach the shared blob to sd_asym_cpucapacity only when asym is a non-overlapping domain, otherwise fallback to sd_llc and, in this case, ignore has_idle_cores in select_idle_capacity(). This might be not the best in terms of efficiency on those exotic topologies, but it'd eliminate the overlap/aliasing risk, while still being correct. What do you think?
Thanks,
-Andrea
Re: [PATCH 1/2] sched/fair: Prefer fully-idle SMT cores in asym-capacity idle selection
Hi Prateek,
On Mon, Apr 20, 2026 at 11:42:23PM +0200, Andrea Righi wrote:
...
Quoted 49 lines
Quoted 41 lines
Quoted 10 lines
Quoted 9 lines
I still have one question: Can first
SD_ASYM_CPUCAPACITY_FULLbe set at aSD_NUMA?We'll need to deal with overlapping domains then but seems like it could be possible with weird cpusets :-(
But in that case, do we even want to search CPUs outside the NUMA in
select_idle_capacity()? I don't think anything stops this currently but I might be wrong.Quoted 7 lines
My $0.02 on this.
In theory it could happen with unusual topologies or constrained cpusets, although it should be quite rare. That said,
select_idle_capacity()already operates on the span ofsd_asym_cpucapacity, so if that domain crosses NUMA boundaries, we're already scanning across NUMA today. This patch doesn't fundamentally alter this behavior.Ack! I was just thinking loud from the topology standpoint since
sd->sharedis not designed to handle the overlapping domains likesg->sgcdoes but we can probably figure some way to make it work.Using the ring topology example from
topology.c:0 ----- 1
| | | | | |3 ----- 2
Consider NUMA-1 below gets the
SD_ASYM_CPUCAPACITY_FULLflag:NUMA-2 0-3 0-3 0-3 0-3 groups: {0-1,3},{1-3} {0-2},{0,2-3} {1-3},{0-1,3} {0,2-3},{0-2}NUMA-1 0-1,3 0-2 1-3 0,2-3 groups: {0},{1},{3} {0},{1},{2} {1},{2},{3} {0},{2},{3}NUMA-0 0 1 2 3
Quoted 7 lines
The "
sd->shared" assignments at NUMA-1 will put first, second, and the last domain in the same "shared" range by today's logic since the first CPU in their span is the same although their spans are slightly different.The third will be standalone since the first CPU of the domain span will be different.
Quoted 6 lines
Yeah, makes sense. I'm wondering if we should attach the shared blob to
sd_asym_cpucapacityonly when asym is a non-overlapping domain, otherwise fallback tosd_llcand, in this case, ignorehas_idle_coresinselect_idle_capacity(). This might be not the best in terms of efficiency on those exotic topologies, but it'd eliminate the overlap/aliasing risk, while still being correct. What do you think?
I slightly changed your patch adding this logic on top, I'll send an updated patch series, so it's easier to review/comment.
Thanks,
-Andrea
Re: [PATCH 1/2] sched/fair: Prefer fully-idle SMT cores in asym-capacity idle selection
Quoted 60 lines · Andrea Righi
Hi Prateek,
On Mon, Apr 20, 2026 at 11:42:23PM +0200, Andrea Righi wrote:
...Quoted 49 lines
Quoted 41 lines
Quoted 10 lines
Quoted 9 lines
I still have one question: Can first
SD_ASYM_CPUCAPACITY_FULLbe set at aSD_NUMA?We'll need to deal with overlapping domains then but seems like it could be possible with weird cpusets :-(
But in that case, do we even want to search CPUs outside the NUMA in
select_idle_capacity()? I don't think anything stops this currently but I might be wrong.Quoted 7 lines
My $0.02 on this.
In theory it could happen with unusual topologies or constrained cpusets, although it should be quite rare. That said,
select_idle_capacity()already operates on the span ofsd_asym_cpucapacity, so if that domain crosses NUMA boundaries, we're already scanning across NUMA today. This patch doesn't fundamentally alter this behavior.Ack! I was just thinking loud from the topology standpoint since
sd->sharedis not designed to handle the overlapping domains likesg->sgcdoes but we can probably figure some way to make it work.Using the ring topology example from
topology.c:0 ----- 1
| | | | | |3 ----- 2
Consider NUMA-1 below gets the
SD_ASYM_CPUCAPACITY_FULLflag:NUMA-2 0-3 0-3 0-3 0-3 groups: {0-1,3},{1-3} {0-2},{0,2-3} {1-3},{0-1,3} {0,2-3},{0-2}NUMA-1 0-1,3 0-2 1-3 0,2-3 groups: {0},{1},{3} {0},{1},{2} {1},{2},{3} {0},{2},{3}NUMA-0 0 1 2 3
Quoted 7 lines
The "
sd->shared" assignments at NUMA-1 will put first, second, and the last domain in the same "shared" range by today's logic since the first CPU in their span is the same although their spans are slightly different.The third will be standalone since the first CPU of the domain span will be different.
Quoted 6 lines
Yeah, makes sense. I'm wondering if we should attach the shared blob to
sd_asym_cpucapacityonly when asym is a non-overlapping domain, otherwise fallback tosd_llcand, in this case, ignorehas_idle_coresinselect_idle_capacity(). This might be not the best in terms of efficiency on those exotic topologies, but it'd eliminate the overlap/aliasing risk, while still being correct. What do you think?
Quoted 2 lines
I slightly changed your patch adding this logic on top, I'll send an updated patch series, so it's easier to review/comment.
Actually... while preparing the series I realized that in select_idle_capacity() we may end up clearing the has_idle_cores hint even when the failure is due to affinity constraints (no fit CPU in the allowed cpumask), not only when no fully idle core is found in the system and this can lead to false has_idle_cores hints.
At this point I'm wondering if it's better to just ignore the has_idle_cores hint completely in the smt+asym-cpu-capacity scenario (which would also simplify the exotic topology cases).
I did some quick tests with this on Vera and I'm getting pretty much the same performance results. Opinions? Am I missing something?
Thanks,
-Andrea
K Prateek Nayak
reviewerRe: [PATCH 1/2] sched/fair: Prefer fully-idle SMT cores in asym-capacity idle selection
Hello Andrea,
Quoted 5 lines · Andrea Righi
Actually... while preparing the series I realized that in
select_idle_capacity()we may end up clearing thehas_idle_coreshint even when the failure is due to affinity constraints (no fit CPU in the allowed cpumask), not only when no fully idle core is found in the system and this can lead to falsehas_idle_coreshints.
This is also the case with select_idle_cpu() but any core turning idle will again reset the indicator so it should be fine for most part where there is a lot of blocking + wakeup.
Quoted 7 lines
At this point I'm wondering if it's better to just ignore the
has_idle_coreshint completely in the smt+asym-cpu-capacity scenario (which would also simplify the exotic topology cases).I did some quick tests with this on Vera and I'm getting pretty much the same performance results. Opinions? Am I missing something?
I don't think so. Generally it is counterproductive to search a lot in a busy system but I guess just making the path SMT aware give a much better result compared to the baseline that it doesn't matter.
Can I trouble you to test the SIS_UTIL bailout with your series + the topology changes:
kernel/sched/fair.c +21 -0
@@ -7990,6 +7990,7 @@ select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int target)int fits, best_fits =0 ;int cpu, best_cpu =-1 ;struct cpumask *cpus; +int nr =INT_MAX ; cpus =this_cpu_cpumask_var_ptr (select_rq_mask);cpumask_and (cpus,sched_domain_span (sd), p->cpus_ptr);
@@ -7998,10 +7999,30 @@ select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int target) util_min =uclamp_eff_value (p,UCLAMP_MIN ); util_max =uclamp_eff_value (p,UCLAMP_MAX ); +if (sched_feat (SIS_UTIL ) && sd->shared) { +/* +* Increment because !--nr is the condition to stop scan. +* +* Since "sd" is "sd_llc" for target CPU dereferenced in the +* caller, it is safe to directly dereference "sd->shared". +* Topology bits always ensure it assigned for "sd_llc" abd it +* cannot disappear as long as we have a RCU protected +* reference to one the associated "sd" here. +*/ + nr =READ_ONCE (sd->shared->nr_idle_scan) +1 ; +/* overloaded LLC is unlikely to have idle cpu/core */ +if (nr ==1 ) +return -1 ; + } +for_each_cpu_wrap (cpu, cpus, target) {bool preferred_core = !prefers_idle_core ||is_core_idle (cpu);unsigned long cpu_cap =capacity_of (cpu); +/* We have found a good enough target. Just use it. */ +if (--nr <=0 && best_fits ==-4 ) +return best_cpu; +if (!choose_idle_cpu (cpu, p))continue ;
---
You can also try "best_fits <= -3" in that last bailout condition and see if that help.
Signature · 2 lines
Thanks and Regards, Prateek
Re: [PATCH 1/2] sched/fair: Prefer fully-idle SMT cores in asym-capacity idle selection
Quoted 9 lines · K Prateek Nayak
Hello Andrea,
Quoted 5 lines · Andrea Righi
Actually... while preparing the series I realized that in
select_idle_capacity()we may end up clearing thehas_idle_coreshint even when the failure is due to affinity constraints (no fit CPU in the allowed cpumask), not only when no fully idle core is found in the system and this can lead to falsehas_idle_coreshints.
Quoted 3 lines
This is also the case with
select_idle_cpu()but any core turning idle will again reset the indicator so it should be fine for most part where there is a lot of blocking + wakeup.
You're right, select_idle_cpu() is also iterating on the allowed CPUs. So, nevermind, we definitely want to use the has_idle_cores hint.
Quoted 61 lines
Quoted 8 lines
At this point I'm wondering if it's better to just ignore the
has_idle_coreshint completely in the smt+asym-cpu-capacity scenario (which would also simplify the exotic topology cases).I did some quick tests with this on Vera and I'm getting pretty much the same performance results. Opinions? Am I missing something?
I don't think so. Generally it is counterproductive to search a lot in a busy system but I guess just making the path SMT aware give a much better result compared to the baseline that it doesn't matter.
Can I trouble you to test the
SIS_UTILbailout with your series + the topology changes:kernel/sched/fair.c +21
-1@@ -7990,6 +7990,7 @@ select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int target)int fits, best_fits =0 ;int cpu, best_cpu =-1 ;struct cpumask *cpus; +int nr =INT_MAX ; cpus =this_cpu_cpumask_var_ptr (select_rq_mask);cpumask_and (cpus,sched_domain_span (sd), p->cpus_ptr);@@ -7998,10 +7999,30 @@ select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int target) util_min =uclamp_eff_value (p,UCLAMP_MIN ); util_max =uclamp_eff_value (p,UCLAMP_MAX ); +if (sched_feat (SIS_UTIL ) && sd->shared) { +/* +* Increment because !--nr is the condition to stop scan. +* +* Since "sd" is "sd_llc" for target CPU dereferenced in the +* caller, it is safe to directly dereference "sd->shared". +* Topology bits always ensure it assigned for "sd_llc" abd it +* cannot disappear as long as we have a RCU protected +* reference to one the associated "sd" here. +*/ + nr =READ_ONCE (sd->shared->nr_idle_scan) +1 ; +/* overloaded LLC is unlikely to have idle cpu/core */ +if (nr ==1 ) +return -1 ; + } +for_each_cpu_wrap (cpu, cpus, target) {bool preferred_core = !prefers_idle_core ||is_core_idle (cpu);unsigned long cpu_cap =capacity_of (cpu); +/* We have found a good enough target. Just use it. */ +if (--nr <=0 && best_fits ==-4 ) +return best_cpu; +if (!choose_idle_cpu (cpu, p))continue ; ---
Quoted 2 lines
You can also try "
best_fits<= -3" in that last bailout condition and see if that help.
Sure, will test in a bit!
Thanks,
-Andrea
Re: [PATCH 1/2] sched/fair: Prefer fully-idle SMT cores in asym-capacity idle selection
In reply to K Prateek NayakHi Prateek,
On Tue, Apr 21, 2026 at 04:52:46PM +0530, K Prateek Nayak wrote:
...
Quoted 16 lines
Can I trouble you to test the
SIS_UTILbailout with your series + the topology changes:kernel/sched/fair.c +1
-0@@ -7990,6 +7990,7 @@ select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int target)int fits, best_fits =0 ;int cpu, best_cpu =-1 ;struct cpumask *cpus; +int nr =INT_MAX ; cpus =this_cpu_cpumask_var_ptr (select_rq_mask);cpumask_and (cpus,sched_domain_span (sd), p->cpus_ptr);
kernel/sched/fair.c select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int target) · line 8008
Quoted 10 lines
kernel/sched/fair.c +7
-0@@ -7998,10 +7999,30 @@ select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int target) util_min =uclamp_eff_value (p,UCLAMP_MIN ); util_max =uclamp_eff_value (p,UCLAMP_MAX ); +if (sched_feat (SIS_UTIL ) && sd->shared) { +/* +* Increment because !--nr is the condition to stop scan. +* +* Since "sd" is "sd_llc" for target CPU dereferenced in the +* caller, it is safe to directly dereference "sd->shared". +* Topology bits always ensure it assigned for "sd_llc" abd it
nit: s/abd/and/, BTW we have the same in select_idle_cpu(), consistent typo. :)
Quoted 22 lines
kernel/sched/fair.c +13
-1+* cannot disappear as long as we have a RCU protected +* reference to one the associated "sd" here. +*/ + nr =READ_ONCE (sd->shared->nr_idle_scan) +1 ; +/* overloaded LLC is unlikely to have idle cpu/core */ +if (nr ==1 ) +return -1 ; + } +for_each_cpu_wrap (cpu, cpus, target) {bool preferred_core = !prefers_idle_core ||is_core_idle (cpu);unsigned long cpu_cap =capacity_of (cpu); +/* We have found a good enough target. Just use it. */ +if (--nr <=0 && best_fits ==-4 ) +return best_cpu; +if (!choose_idle_cpu (cpu, p))continue ; ---
Quoted 2 lines
You can also try "
best_fits<= -3" in that last bailout condition and see if that help.
For the bailout condition I don't see much difference using either <= -3 or == -4. In general, I see a small but consistent improvement with the SIS_UTIL logic, especially when the system is close to saturation (as expected).
So, this looks good to me! Do you want me to include also this one in the new SMT-aware asym cpu capacity patch series (keeping your authorship of course) or do you prefer to route this separately?
Thanks,
-Andrea
K Prateek Nayak
reviewerRe: [PATCH 1/2] sched/fair: Prefer fully-idle SMT cores in asym-capacity idle selection
Hello Andrea,
Quoted 6 lines · Andrea Righi
Quoted 2 lines
You can also try "
best_fits<= -3" in that last bailout condition and see if that help.For the bailout condition I don't see much difference using either <= -3 or == -4. In general, I see a small but consistent improvement with the
SIS_UTILlogic, especially when the system is close to saturation (as expected).
Thank you for testing! I guess == -4 is safer then.
It is probably best to add an enum of sorts to help distinguish these states rather than the magic numbers. Perhaps something like:
enum asym_fits_state {
/* In descending order of preference */
ASYM_IDLE_CORE_UCLAMP_MISFIT = -4,
ASYM_IDLE_CORE_COMPLETE_MISFIT,
ASYM_IDLE_THREAD_FITS,
ASYM_IDLE_THREAD_UCLAMP_MISFIT,
ASYM_IDLE_COMPLETE_MISFIT,
/* asym_fits_cpu() bias for an idle core. */
ASYM_IDLE_CORE_BIAS = -3,
};Quoted 4 lines
So, this looks good to me! Do you want me to include also this one in the new SMT-aware asym cpu capacity patch series (keeping your authorship of course) or do you prefer to route this separately?
I think you can send it as a part of your series for easy review. I'll be happy to help reworking those bits based on the comments if folks aren't happy with them ;-)
Signature · 2 lines
Thanks and Regards, Prateek
Re: [PATCH 1/2] sched/fair: Prefer fully-idle SMT cores in asym-capacity idle selection
Quoted 28 lines · K Prateek Nayak
Hello Andrea,
Quoted 6 lines · Andrea Righi
Quoted 2 lines
You can also try "
best_fits<= -3" in that last bailout condition and see if that help.For the bailout condition I don't see much difference using either <= -3 or == -4. In general, I see a small but consistent improvement with the
SIS_UTILlogic, especially when the system is close to saturation (as expected).Thank you for testing! I guess == -4 is safer then.
It is probably best to add an enum of sorts to help distinguish these states rather than the magic numbers. Perhaps something like:
enum asym_fits_state { /* In descending order of preference */ ASYM_IDLE_CORE_UCLAMP_MISFIT = -4, ASYM_IDLE_CORE_COMPLETE_MISFIT, ASYM_IDLE_THREAD_FITS, ASYM_IDLE_THREAD_UCLAMP_MISFIT, ASYM_IDLE_COMPLETE_MISFIT, /* asym_fits_cpu() bias for an idle core. */ ASYM_IDLE_CORE_BIAS = -3, };Quoted 1 line
Quoted 7 lines
Quoted 3 lines
So, this looks good to me! Do you want me to include also this one in the new SMT-aware asym cpu capacity patch series (keeping your authorship of course) or do you prefer to route this separately?
I think you can send it as a part of your series for easy review. I'll be happy to help reworking those bits based on the comments if folks aren't happy with them ;-)
BTW, the SIS_UTIL part also improves performance on Grace (tested also there to make sure we were not regressing the non-SMT asym-cpu-capacity case).
So, definitely +1 to include this from me. :)
Thanks,
-Andrea
Vincent Guittot
maintainerRe: [PATCH 1/2] sched/fair: Prefer fully-idle SMT cores in asym-capacity idle selection
In reply to Andrea RighiQuoted 62 lines · Andrea Righi
Quoted 60 lines · Andrea Righi
Hi Prateek,
On Mon, Apr 20, 2026 at 11:42:23PM +0200, Andrea Righi wrote:
...Quoted 55 lines
Quoted 41 lines
Quoted 10 lines
Quoted 9 lines
I still have one question: Can first
SD_ASYM_CPUCAPACITY_FULLbe set at aSD_NUMA?We'll need to deal with overlapping domains then but seems like it could be possible with weird cpusets :-(
But in that case, do we even want to search CPUs outside the NUMA in
select_idle_capacity()? I don't think anything stops this currently but I might be wrong.Quoted 7 lines
My $0.02 on this.
In theory it could happen with unusual topologies or constrained cpusets, although it should be quite rare. That said,
select_idle_capacity()already operates on the span ofsd_asym_cpucapacity, so if that domain crosses NUMA boundaries, we're already scanning across NUMA today. This patch doesn't fundamentally alter this behavior.Ack! I was just thinking loud from the topology standpoint since
sd->sharedis not designed to handle the overlapping domains likesg->sgcdoes but we can probably figure some way to make it work.Using the ring topology example from
topology.c:0 ----- 1
| | | | | |3 ----- 2
Consider NUMA-1 below gets the
SD_ASYM_CPUCAPACITY_FULLflag:NUMA-2 0-3 0-3 0-3 0-3 groups: {0-1,3},{1-3} {0-2},{0,2-3} {1-3},{0-1,3} {0,2-3},{0-2}NUMA-1 0-1,3 0-2 1-3 0,2-3 groups: {0},{1},{3} {0},{1},{2} {1},{2},{3} {0},{2},{3}NUMA-0 0 1 2 3
Quoted 7 lines
The "
sd->shared" assignments at NUMA-1 will put first, second, and the last domain in the same "shared" range by today's logic since the first CPU in their span is the same although their spans are slightly different.The third will be standalone since the first CPU of the domain span will be different.
Yeah, makes sense. I'm wondering if we should attach the shared blob to
sd_asym_cpucapacityonly when asym is a non-overlapping domain, otherwise fallback tosd_llcand, in this case, ignorehas_idle_coresinselect_idle_capacity(). This might be not the best in terms of efficiency on those exotic topologies, but it'd eliminate the overlap/aliasing risk, while still being correct. What do you think?
Quoted 8 lines
Quoted 2 lines
I slightly changed your patch adding this logic on top, I'll send an updated patch series, so it's easier to review/comment.
Actually... while preparing the series I realized that in
select_idle_capacity()we may end up clearing thehas_idle_coreshint even when the failure is due to affinity constraints (no fit CPU in the allowed cpumask), not only when no fully idle core is found in the system and this can lead to falsehas_idle_coreshints.
How is it different from select_idle_cpu() which does the same afaict ?
Quoted 10 lines
At this point I'm wondering if it's better to just ignore the
has_idle_coreshint completely in the smt+asym-cpu-capacity scenario (which would also simplify the exotic topology cases).I did some quick tests with this on Vera and I'm getting pretty much the same performance results. Opinions? Am I missing something?
Thanks,
-Andrea
Re: [PATCH 1/2] sched/fair: Prefer fully-idle SMT cores in asym-capacity idle selection
On Tue, Apr 21, 2026 at 02:26:58PM +0200, Vincent Guittot wrote:
...
Quoted 10 lines
Quoted 10 lines
Quoted 9 lines
Quoted 6 lines
Yeah, makes sense. I'm wondering if we should attach the shared blob to
sd_asym_cpucapacityonly when asym is a non-overlapping domain, otherwise fallback tosd_llcand, in this case, ignorehas_idle_coresinselect_idle_capacity(). This might be not the best in terms of efficiency on those exotic topologies, but it'd eliminate the overlap/aliasing risk, while still being correct. What do you think?I slightly changed your patch adding this logic on top, I'll send an updated patch series, so it's easier to review/comment.
Quoted 7 lines
Quoted 5 lines
Actually... while preparing the series I realized that in
select_idle_capacity()we may end up clearing thehas_idle_coreshint even when the failure is due to affinity constraints (no fit CPU in the allowed cpumask), not only when no fully idle core is found in the system and this can lead to falsehas_idle_coreshints.How is it different from
select_idle_cpu()which does the same afaict ?
Nah, it was just me not reading the code right. It's the same with select_idle_cpu(), so I'm back at using Prateek's patch.
Thanks,
-Andrea
Vincent Guittot
maintainerRe: [PATCH 1/2] sched/fair: Prefer fully-idle SMT cores in asym-capacity idle selection
In reply to Andrea RighiQuoted 92 lines · Andrea Righi
On systems with asymmetric CPU capacity (e.g., ACPI/CPPC reporting different per-core frequencies), the wakeup path uses
select_idle_capacity()and prioritizes idle CPUs with higher capacity for better task placement.However, when those CPUs belong to SMT cores, their effective capacity can be much lower than the nominal capacity when the sibling thread is
busy: SMT siblings compete for shared resources, so a "high capacity" CPU that is idle but whose sibling is busy does not deliver its full capacity. This effective capacity reduction cannot be modeled by the static capacity value alone.When SMT is active, teach asym-capacity idle selection to treat a logical CPU as a weaker target if its physical core is only partially
idle:select_idle_capacity()no longer returns on the first idle CPU whose static capacity fits the task when that CPU still has a busy sibling, it keeps scanning for an idle CPU on a fully-idle core and only if none qualify does it fall back to partially-idle cores, using shifted fit scores so fully-idle cores win ties;asym_fits_cpu()applies the same fully-idle core requirement when asym capacity and SMT are both active.This improves task placement, since partially-idle SMT siblings deliver less than their nominal capacity. Favoring fully idle cores, when available, can significantly enhance both throughput and wakeup latency on systems with both SMT and CPU asymmetry.
No functional changes on systems with only asymmetric CPUs or only SMT.
Cc K Prateek Nayak <[email protected]>Cc Vincent Guittot <[email protected]>Cc Dietmar Eggemann <[email protected]>Cc Christian Loehle <[email protected]>Cc Koba Ko <[email protected]>Reported-by Felix Abecassis <[email protected]>Signed-off-by Andrea Righi <[email protected]>--- kernel/sched/fair.c | 36 ++++++++++++++++++++++++++++++++---- 1 file changed, 32 insertions(+), 4 deletions(-)kernel/sched/fair.c +21
-1@@ -7774,6 +7774,7 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, boolstatic int select_idle_capacity (struct task_struct *p,struct sched_domain *sd,int target) { +bool prefers_idle_core =sched_smt_active () &&test_idle_cores (target);unsigned long task_util, util_min, util_max, best_cap =0 ;int fits, best_fits =0 ;int cpu, best_cpu =-1 ;@@ -7787,6 +7788,7 @@ select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int target) util_max =uclamp_eff_value (p,UCLAMP_MAX );for_each_cpu_wrap (cpu, cpus, target) { +bool preferred_core = !prefers_idle_core ||is_core_idle (cpu);unsigned long cpu_cap =capacity_of (cpu);if (!available_idle_cpu (cpu) && !sched_idle_cpu (cpu))@@ -7795,7 +7797,7 @@ select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int target) fits =util_fits_cpu (task_util, util_min, util_max, cpu);/* This CPU fits with all requirements */ -if (fits >0 ) +if (fits >0 && preferred_core)return cpu;/* * Only the min performance hint (i.e. uclamp_min) doesn't fit. @@ -7803,9 +7805,30 @@ select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int target)*/ else if (fits < 0) cpu_cap =get_actual_cpu_capacity (cpu); +/* +* fits > 0 implies we are not on a preferred core +* but the util fits CPU capacity. Set fits to -2 so +* the effective range becomes [-2, 0] where: +* 0 - does not fit +* -1 - fits with the exception of UCLAMP_MIN +* -2 - fits with the exception of preferred_core +*/ + else if (fits > 0) + fits =-2 ; + +/* +* If we are on a preferred core, translate the range of fits +* of [-1, 0] to [-4, -3]. This ensures that an idle core +* is always given priority over (partially) busy core. +* +* A fully fitting idle core would have returned early and hence +* fits > 0 for preferred_core need not be dealt with.
kernel/sched/fair.c select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int target) · line 7834
Quoted 10 lines
kernel/sched/fair.c +4
-1+*/ +if (preferred_core) + fits -=3 ;/* -* First, select CPU which fits better (-1 being better than 0). +* First, select CPU which fits better (lower is more preferred). * Then, select the one with best capacity at same level. */ if ((fits < best_fits) ||
You have to clear idle_core if you were looking of an idle core but didn't find one while looping on CPUs.
You need the following to clear idle core:
@@ -7739,6 +7739,11 @@ select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int target)
}
}+ /* The range [-4, -3] implies at least one idle core, the values above
+ * imply that we didn't find anyone while looping CPUs */
+ if (prefers_idle_core && fits > -3)
+ set_idle_cores(target, false);
+
return best_cpu;
}
Quoted 23 lines
kernel/sched/fair.c +7
-3@@ -7824,12 +7847,17 @@ static inline bool asym_fits_cpu(unsigned long util, unsigned long util_max, int cpu) { - if (sched_asym_cpucap_active()) + if (sched_asym_cpucap_active()) {/* * Return true only if the cpu fully fits the task requirements * which include the utilization and the performance hints. +* +* When SMT is active, also require that the core has no busy +* siblings. */ - return (util_fits_cpu(util, util_min, util_max, cpu) > 0); +return (!sched_smt_active () ||is_core_idle (cpu)) && + (util_fits_cpu (util, util_min, util_max, cpu) >0 ); + }return true; } --2.53.0
Re: [PATCH 1/2] sched/fair: Prefer fully-idle SMT cores in asym-capacity idle selection
Hi Vincent,
Quoted 111 lines · Vincent Guittot
Quoted 92 lines · Andrea Righi
On systems with asymmetric CPU capacity (e.g., ACPI/CPPC reporting different per-core frequencies), the wakeup path uses
select_idle_capacity()and prioritizes idle CPUs with higher capacity for better task placement.However, when those CPUs belong to SMT cores, their effective capacity can be much lower than the nominal capacity when the sibling thread is
busy: SMT siblings compete for shared resources, so a "high capacity" CPU that is idle but whose sibling is busy does not deliver its full capacity. This effective capacity reduction cannot be modeled by the static capacity value alone.When SMT is active, teach asym-capacity idle selection to treat a logical CPU as a weaker target if its physical core is only partially
idle:select_idle_capacity()no longer returns on the first idle CPU whose static capacity fits the task when that CPU still has a busy sibling, it keeps scanning for an idle CPU on a fully-idle core and only if none qualify does it fall back to partially-idle cores, using shifted fit scores so fully-idle cores win ties;asym_fits_cpu()applies the same fully-idle core requirement when asym capacity and SMT are both active.This improves task placement, since partially-idle SMT siblings deliver less than their nominal capacity. Favoring fully idle cores, when available, can significantly enhance both throughput and wakeup latency on systems with both SMT and CPU asymmetry.
No functional changes on systems with only asymmetric CPUs or only SMT.
Cc K Prateek Nayak <[email protected]>Cc Vincent Guittot <[email protected]>Cc Dietmar Eggemann <[email protected]>Cc Christian Loehle <[email protected]>Cc Koba Ko <[email protected]>Reported-by Felix Abecassis <[email protected]>Signed-off-by Andrea Righi <[email protected]>--- kernel/sched/fair.c | 36 ++++++++++++++++++++++++++++++++---- 1 file changed, 32 insertions(+), 4 deletions(-)kernel/sched/fair.c +21
-1@@ -7774,6 +7774,7 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, boolstatic int select_idle_capacity (struct task_struct *p,struct sched_domain *sd,int target) { +bool prefers_idle_core =sched_smt_active () &&test_idle_cores (target);unsigned long task_util, util_min, util_max, best_cap =0 ;int fits, best_fits =0 ;int cpu, best_cpu =-1 ;@@ -7787,6 +7788,7 @@ select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int target) util_max =uclamp_eff_value (p,UCLAMP_MAX );for_each_cpu_wrap (cpu, cpus, target) { +bool preferred_core = !prefers_idle_core ||is_core_idle (cpu);unsigned long cpu_cap =capacity_of (cpu);if (!available_idle_cpu (cpu) && !sched_idle_cpu (cpu))@@ -7795,7 +7797,7 @@ select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int target) fits =util_fits_cpu (task_util, util_min, util_max, cpu);/* This CPU fits with all requirements */ -if (fits >0 ) +if (fits >0 && preferred_core)return cpu;/* * Only the min performance hint (i.e. uclamp_min) doesn't fit. @@ -7803,9 +7805,30 @@ select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int target)*/ else if (fits < 0) cpu_cap =get_actual_cpu_capacity (cpu); +/* +* fits > 0 implies we are not on a preferred core +* but the util fits CPU capacity. Set fits to -2 so +* the effective range becomes [-2, 0] where: +* 0 - does not fit +* -1 - fits with the exception of UCLAMP_MIN +* -2 - fits with the exception of preferred_core +*/ + else if (fits > 0) + fits =-2 ; + +/* +* If we are on a preferred core, translate the range of fits +* of [-1, 0] to [-4, -3]. This ensures that an idle core +* is always given priority over (partially) busy core. +* +* A fully fitting idle core would have returned early and hence +* fits > 0 for preferred_core need not be dealt with. kernel/sched/fair.c select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int target) · line 7834
Quoted 10 lines
kernel/sched/fair.c +4
-1+*/ +if (preferred_core) + fits -=3 ;/* -* First, select CPU which fits better (-1 being better than 0). +* First, select CPU which fits better (lower is more preferred). * Then, select the one with best capacity at same level. */ if ((fits < best_fits) ||You have to clear
idle_coreif you were looking of an idle core but didn't find one while looping on CPUs.You need the following to clear idle core:
(quoted diff) +0
-0@@ -7739,6 +7739,11 @@ select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int target)
Review select_idle_capacity(struct task_struct *p, · line 7749
Quoted 10 lines
(quoted diff) +5
-0} } + /* The range [-4, -3] implies at least one idle core, the values above + * imply that we didn't find anyone while looping CPUs */ + if (prefers_idle_core && fits > -3) + set_idle_cores(target, false); + return best_cpu; }
That makes sense! But it should be best_fits instead of fits, right?
Thanks,
-Andrea
Vincent Guittot
maintainerRe: [PATCH 1/2] sched/fair: Prefer fully-idle SMT cores in asym-capacity idle selection
Quoted 118 lines · Andrea Righi
Hi Vincent,
Quoted 114 lines · Vincent Guittot
Quoted 92 lines · Andrea Righi
On systems with asymmetric CPU capacity (e.g., ACPI/CPPC reporting different per-core frequencies), the wakeup path uses
select_idle_capacity()and prioritizes idle CPUs with higher capacity for better task placement.However, when those CPUs belong to SMT cores, their effective capacity can be much lower than the nominal capacity when the sibling thread is
busy: SMT siblings compete for shared resources, so a "high capacity" CPU that is idle but whose sibling is busy does not deliver its full capacity. This effective capacity reduction cannot be modeled by the static capacity value alone.When SMT is active, teach asym-capacity idle selection to treat a logical CPU as a weaker target if its physical core is only partially
idle:select_idle_capacity()no longer returns on the first idle CPU whose static capacity fits the task when that CPU still has a busy sibling, it keeps scanning for an idle CPU on a fully-idle core and only if none qualify does it fall back to partially-idle cores, using shifted fit scores so fully-idle cores win ties;asym_fits_cpu()applies the same fully-idle core requirement when asym capacity and SMT are both active.This improves task placement, since partially-idle SMT siblings deliver less than their nominal capacity. Favoring fully idle cores, when available, can significantly enhance both throughput and wakeup latency on systems with both SMT and CPU asymmetry.
No functional changes on systems with only asymmetric CPUs or only SMT.
Cc K Prateek Nayak <[email protected]>Cc Vincent Guittot <[email protected]>Cc Dietmar Eggemann <[email protected]>Cc Christian Loehle <[email protected]>Cc Koba Ko <[email protected]>Reported-by Felix Abecassis <[email protected]>Signed-off-by Andrea Righi <[email protected]>--- kernel/sched/fair.c | 36 ++++++++++++++++++++++++++++++++---- 1 file changed, 32 insertions(+), 4 deletions(-)kernel/sched/fair.c +21
-1@@ -7774,6 +7774,7 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, boolstatic int select_idle_capacity (struct task_struct *p,struct sched_domain *sd,int target) { +bool prefers_idle_core =sched_smt_active () &&test_idle_cores (target);unsigned long task_util, util_min, util_max, best_cap =0 ;int fits, best_fits =0 ;int cpu, best_cpu =-1 ;@@ -7787,6 +7788,7 @@ select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int target) util_max =uclamp_eff_value (p,UCLAMP_MAX );for_each_cpu_wrap (cpu, cpus, target) { +bool preferred_core = !prefers_idle_core ||is_core_idle (cpu);unsigned long cpu_cap =capacity_of (cpu);if (!available_idle_cpu (cpu) && !sched_idle_cpu (cpu))@@ -7795,7 +7797,7 @@ select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int target) fits =util_fits_cpu (task_util, util_min, util_max, cpu);/* This CPU fits with all requirements */ -if (fits >0 ) +if (fits >0 && preferred_core)return cpu;/* * Only the min performance hint (i.e. uclamp_min) doesn't fit. @@ -7803,9 +7805,30 @@ select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int target)*/ else if (fits < 0) cpu_cap =get_actual_cpu_capacity (cpu); +/* +* fits > 0 implies we are not on a preferred core +* but the util fits CPU capacity. Set fits to -2 so +* the effective range becomes [-2, 0] where: +* 0 - does not fit +* -1 - fits with the exception of UCLAMP_MIN +* -2 - fits with the exception of preferred_core +*/ + else if (fits > 0) + fits =-2 ; + +/* +* If we are on a preferred core, translate the range of fits +* of [-1, 0] to [-4, -3]. This ensures that an idle core +* is always given priority over (partially) busy core. +* +* A fully fitting idle core would have returned early and hence +* fits > 0 for preferred_core need not be dealt with. kernel/sched/fair.c select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int target) · line 7834
Quoted 10 lines
kernel/sched/fair.c +4
-1+*/ +if (preferred_core) + fits -=3 ;/* -* First, select CPU which fits better (-1 being better than 0). +* First, select CPU which fits better (lower is more preferred). * Then, select the one with best capacity at same level. */ if ((fits < best_fits) ||You have to clear
idle_coreif you were looking of an idle core but didn't find one while looping on CPUs.You need the following to clear idle core:
(quoted diff) +0
-0@@ -7739,6 +7739,11 @@ select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int target) } }
Quoted 9 lines
Review select_idle_capacity(struct task_struct *p, · line 7749
Quoted 7 lines
(quoted diff) +5
-0+ /* The range [-4, -3] implies at least one idle core, the values above + * imply that we didn't find anyone while looping CPUs */ + if (prefers_idle_core && fits > -3) + set_idle_cores(target, false); + return best_cpu; }That makes sense! But it should be
best_fitsinstead of fits, right?
yes, of course
Quoted 3 lines
Thanks,
-Andrea
PATCH v2 2/2 sched/fair: Reject misfit pulls onto busy SMT siblings on asym-capacity
In reply to Andrea RighiWhen SD_ASYM_CPUCAPACITY load balancing considers pulling a misfit task, capacity_of(dst_cpu) can overstate available compute if the SMT sibling is busy: the core does not deliver its full nominal capacity.
If SMT is active and dst_cpu is not on a fully idle core, skip this destination so we do not migrate a misfit expecting a capacity upgrade we cannot actually provide.
No functional changes on systems with only asymmetric CPUs or only SMT.
--- kernel/sched/fair.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-)
kernel/sched/fair.c +7 -1
@@ -10607,10 +10607,16 @@ static bool update_sd_pick_busiest(struct lb_env *env,* We can use max_capacity here as reduction in capacity on some * CPUs in the group should either be possible to resolve * internally or be covered by avg_load imbalance (eventually). +* +* When SMT is active, only pull a misfit to dst_cpu if it is on a +* fully idle core; otherwise the effective capacity of the core is +* reduced and we may not actually provide more capacity than the +* source. */ if ((env->sd->flags &SD_ASYM_CPUCAPACITY ) && (sgs->group_type == group_misfit_task) && - (!capacity_greater (capacity_of (env->dst_cpu), sg->sgc->max_capacity) || + ((sched_smt_active () && !is_core_idle (env->dst_cpu)) || + !capacity_greater (capacity_of (env->dst_cpu), sg->sgc->max_capacity) || sds->local_stat.group_type != group_has_spare))return false;
Signature · 1 line
2.53.0