Skip to contentSkip to conversation

sched/fair: SMT-aware asymmetric CPU capacity

PATCH Superseded Andrea Righi 7 participants 44 messages Mar 26 – Apr 7 Scheduler

v1 of 6 Superseded Latest is v6 →

lore
Share & tools
lore
#0 lore raw

cover sched/fair: SMT-aware asymmetric CPU capacity

This series attempts to improve SD_ASYM_CPUCAPACITY scheduling by introducing SMT awareness.

= Problem =

Nominal per-logical-CPU capacity can overstate usable compute when an SMT sibling is busy, because the physical core doesn't deliver its full nominal capacity. So, several SD_ASYM_CPUCAPACITY paths may pick high capacity CPUs that are not actually good destinations.

= Proposed Solution =

This patch set aligns those paths with a simple rule already used
elsewhere: when SMT is active, prefer fully idle cores and avoid treating partially idle SMT siblings as full-capacity targets where that would mislead load balance.

Patch set summary:

- [PATCH 1/4] sched/fair: Prefer fully-idle SMT cores in asym-capacity idle selection

Prefer fully-idle SMT cores in asym-capacity idle selection. In the wakeup fast path, extend select_idle_capacity() / asym_fits_cpu() so idle selection can prefer CPUs on fully idle cores, with a safe fallback.

- [PATCH 2/4] sched/fair: Reject misfit pulls onto busy SMT siblings on asym-capacity

Reject misfit pulls onto busy SMT siblings on SD_ASYM_CPUCAPACITY. Provided for consistency with PATCH 1/4.

- [PATCH 3/4] sched/fair: Enable EAS with SMT on SD_ASYM_CPUCAPACITY systems

Enable EAS with SD_ASYM_CPUCAPACITY and SMT. Also provided for consistency with PATCH 1/4. I've also tested with/without /proc/sys/kernel/sched_energy_aware enabled (same platform) and haven't noticed any regression.

- [PATCH 4/4] sched/fair: Prefer fully-idle SMT core for NOHZ idle load balancer

When choosing the housekeeping CPU that runs the idle load balancer, prefer an idle CPU on a fully idle core so migrated work lands where effective capacity is available.

The change is still consistent with the same "avoid CPUs with busy sibling" logic and it shows some benefits on Vera, but could have negative impact on other systems, I'm including it for completeness (feedback is appreciated).

This patch set has been tested on the new NVIDIA Vera Rubin platform, where SMT is enabled and the firmware exposes small frequency variations (+/-~5%) as differences in CPU capacity, resulting in SD_ASYM_CPUCAPACITY being set.

Without these patches, performance can drop up to ~2x with CPU-intensive workloads, because the SD_ASYM_CPUCAPACITY idle selection policy does not account for busy SMT siblings.

Alternative approaches have been evaluated, such as equalizing CPU capacities, either by exposing uniform values via firmware (ACPI/CPPC) or normalizing them in the kernel by grouping CPUs within a small capacity window (+-5%) [1][2], or enabling asympacking [3].

However, adding SMT awareness to SD_ASYM_CPUCAPACITY has shown better results so far. Improving this policy also seems worthwhile in general, as other platforms in the future may enable SMT with asymmetric CPU topologies.

[1] https://lore.kernel.org/lkml/[email protected]
[2] https://lore.kernel.org/lkml/[email protected]
[3] https://lore.kernel.org/all/[email protected]/

Andrea Righi (4):
sched/fair: Prefer fully-idle SMT cores in asym-capacity idle selection sched/fair: Reject misfit pulls onto busy SMT siblings on asym-capacity sched/fair: Enable EAS with SMT on SD_ASYM_CPUCAPACITY systems sched/fair: Prefer fully-idle SMT core for NOHZ idle load balancer

 kernel/sched/fair.c     | 163 +++++++++++++++++++++++++++++++++++++++++++-----
 kernel/sched/topology.c |   9 ---
 2 files changed, 147 insertions(+), 25 deletions(-)
#1 lore raw

PATCH 1/4 sched/fair: Prefer fully-idle SMT cores in asym-capacity idle selection

On systems with asymmetric CPU capacity (e.g., ACPI/CPPC reporting different per-core frequencies), the wakeup path uses select_idle_capacity() and prioritizes idle CPUs with higher capacity for better task placement. However, when those CPUs belong to SMT cores, their effective capacity can be much lower than the nominal capacity when the sibling thread is busy: SMT siblings compete for shared resources, so a "high capacity" CPU that is idle but whose sibling is busy does not deliver its full capacity. This effective capacity reduction cannot be modeled by the static capacity value alone.

Introduce SMT awareness in the asym-capacity idle selection policy: when SMT is active prefer fully-idle SMT cores over partially-idle ones. A two-phase selection first tries only CPUs on fully idle cores, then falls back to any idle CPU if none fit.

Prioritizing fully-idle SMT cores yields better task placement because the effective capacity of partially-idle SMT cores is reduced; always preferring them when available leads to more accurate capacity usage on task wakeup.

On an SMT system with asymmetric CPU capacities, SMT-aware idle selection has been shown to improve throughput by around 15-18% for CPU-bound workloads, running an amount of tasks equal to the amount of SMT cores.

Cc Vincent Guittot <[email protected]>
Cc Dietmar Eggemann <[email protected]>
Cc Christian Loehle <[email protected]>
Cc Koba Ko <[email protected]>
---
 kernel/sched/fair.c | 86 +++++++++++++++++++++++++++++++++++++++------
 1 file changed, 75 insertions(+), 11 deletions(-)
kernel/sched/fair.c +75 -11
index d57c02e82f3a1..9a95628669851 100644
@@ -7940,14 +7940,21 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, bool
  * Scan the asym_capacity domain for idle CPUs; pick the first idle one on which
  * the task fits. If no CPU is big enough, but there are idle ones, try to
  * maximize capacity.
+ *
+ * When @prefer_idle_cores is true (asym + SMT and idle cores exist), prefer
+ * CPUs on fully-idle cores over partially-idle ones in a single pass: track
+ * the best candidate among idle-core CPUs and the best among any idle CPU,
+ * then return the idle-core candidate if found, else the best any-idle.
  */
 static int
-select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int target)
+select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int target,
+		     bool prefer_idle_cores)
 {
-	unsigned long task_util, util_min, util_max, best_cap = 0;
-	int fits, best_fits = 0;
-	int cpu, best_cpu = -1;
+	unsigned long task_util, util_min, util_max, best_cap = 0, best_cap_core = 0;
+	int fits, best_fits = 0, best_fits_core = 0;
+	int cpu, best_cpu = -1, best_cpu_core = -1;
 	struct cpumask *cpus;
+	bool on_idle_core;
 
 	cpus = this_cpu_cpumask_var_ptr(select_rq_mask);
 	cpumask_and(cpus, sched_domain_span(sd), p->cpus_ptr);
@@ -7962,16 +7969,58 @@ select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int target)
 		if (!choose_idle_cpu(cpu, p))
 			continue;
 
+		on_idle_core = is_core_idle(cpu);
+		if (prefer_idle_cores && !on_idle_core) {
+			/* Track best among any idle CPU for fallback */
+			fits = util_fits_cpu(task_util, util_min, util_max, cpu);
+			if (fits > 0) {
+				/*
+				 * Full fit: strictly better than fits 0 / -1;
+				 * among several, prefer higher capacity.
+				 */
+				if (best_cpu < 0 || best_fits <= 0 ||
+				    (best_fits > 0 && cpu_cap > best_cap)) {
+					best_cap = cpu_cap;
+					best_cpu = cpu;
+					best_fits = fits;
+				}
+				continue;
+			}
+			if (best_fits > 0)
+				continue;
+			if (fits < 0)
+				cpu_cap = get_actual_cpu_capacity(cpu);
+			if ((fits < best_fits) ||
+			    ((fits == best_fits) && (cpu_cap > best_cap))) {
+				best_cap = cpu_cap;
+				best_cpu = cpu;
+				best_fits = fits;
+			}
+			continue;
+		}
+
 		fits = util_fits_cpu(task_util, util_min, util_max, cpu);
 
 		/* This CPU fits with all requirements */
-		if (fits > 0)
-			return cpu;
+		if (fits > 0) {
+			if (prefer_idle_cores && on_idle_core)
+				return cpu;
+			if (!prefer_idle_cores)
+				return cpu;
+			/*
+			 * Prefer idle cores: record and keep looking for
+			 * idle-core fit.
+			 */
+			best_cap = cpu_cap;
+			best_cpu = cpu;
+			best_fits = fits;
+			continue;
+		}
 		/*
 		 * Only the min performance hint (i.e. uclamp_min) doesn't fit.
 		 * Look for the CPU with best capacity.
 		 */
-		else if (fits < 0)
+		if (fits < 0)
 			cpu_cap = get_actual_cpu_capacity(cpu);
 
 		/*
@@ -7984,8 +8033,17 @@ select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int target)
 			best_cpu = cpu;
 			best_fits = fits;
 		}
+		if (prefer_idle_cores && on_idle_core &&
+		    ((fits < best_fits_core) ||
+		     ((fits == best_fits_core) && (cpu_cap > best_cap_core)))) {
+			best_cap_core = cpu_cap;
+			best_cpu_core = cpu;
+			best_fits_core = fits;
+		}
 	}
 
+	if (prefer_idle_cores && best_cpu_core >= 0)
+		return best_cpu_core;
 	return best_cpu;
 }
 
@@ -7994,12 +8052,17 @@ static inline bool asym_fits_cpu(unsigned long util,
 				 unsigned long util_max,
 				 int cpu)
 {
-	if (sched_asym_cpucap_active())
+	if (sched_asym_cpucap_active()) {
 		/*
 		 * Return true only if the cpu fully fits the task requirements
 		 * which include the utilization and the performance hints.
+		 *
+		 * When SMT is active, also require that the core has no busy
+		 * siblings.
 		 */
-		return (util_fits_cpu(util, util_min, util_max, cpu) > 0);
+		return (!sched_smt_active() || is_core_idle(cpu)) &&
+		       (util_fits_cpu(util, util_min, util_max, cpu) > 0);
+	}
 
 	return true;
 }
@@ -8097,8 +8160,9 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
 		 * capacity path.
 		 */
 		if (sd) {
-			i = select_idle_capacity(p, sd, target);
-			return ((unsigned)i < nr_cpumask_bits) ? i : target;
+			i = select_idle_capacity(p, sd, target,
+				sched_smt_active() && test_idle_cores(target));
+			return ((unsigned int)i < nr_cpumask_bits) ? i : target;
 		}
 	}
 
Signature · 1 line
2.53.0
maintainer
#2 lore raw

Re: [PATCH 1/4] sched/fair: Prefer fully-idle SMT cores in asym-capacity idle selection

Quoted 67 lines · Andrea Righi

On systems with asymmetric CPU capacity (e.g., ACPI/CPPC reporting different per-core frequencies), the wakeup path uses select_idle_capacity() and prioritizes idle CPUs with higher capacity for better task placement. However, when those CPUs belong to SMT cores, their effective capacity can be much lower than the nominal capacity when the sibling thread is busy: SMT siblings compete for shared resources, so a "high capacity" CPU that is idle but whose sibling is busy does not deliver its full capacity. This effective capacity reduction cannot be modeled by the static capacity value alone.

Introduce SMT awareness in the asym-capacity idle selection policy: when SMT is active prefer fully-idle SMT cores over partially-idle ones. A two-phase selection first tries only CPUs on fully idle cores, then falls back to any idle CPU if none fit.

Prioritizing fully-idle SMT cores yields better task placement because the effective capacity of partially-idle SMT cores is reduced; always preferring them when available leads to more accurate capacity usage on task wakeup.

On an SMT system with asymmetric CPU capacities, SMT-aware idle selection has been shown to improve throughput by around 15-18% for CPU-bound workloads, running an amount of tasks equal to the amount of SMT cores.

Cc Vincent Guittot <[email protected]>
Cc Dietmar Eggemann <[email protected]>
Cc Christian Loehle <[email protected]>
Cc Koba Ko <[email protected]>
---
 kernel/sched/fair.c | 86 +++++++++++++++++++++++++++++++++++++++------
 1 file changed, 75 insertions(+), 11 deletions(-)
kernel/sched/fair.c +11 -4
index d57c02e82f3a1..9a95628669851 100644
@@ -7940,14 +7940,21 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, bool
  * Scan the asym_capacity domain for idle CPUs; pick the first idle one on which
  * the task fits. If no CPU is big enough, but there are idle ones, try to
  * maximize capacity.
+ *
+ * When @prefer_idle_cores is true (asym + SMT and idle cores exist), prefer
+ * CPUs on fully-idle cores over partially-idle ones in a single pass: track
+ * the best candidate among idle-core CPUs and the best among any idle CPU,
+ * then return the idle-core candidate if found, else the best any-idle.
  */
 static int
-select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int target)
+select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int target,
+                    bool prefer_idle_cores)
 {
-       unsigned long task_util, util_min, util_max, best_cap = 0;
-       int fits, best_fits = 0;
-       int cpu, best_cpu = -1;
+       unsigned long task_util, util_min, util_max, best_cap = 0, best_cap_core = 0;
+       int fits, best_fits = 0, best_fits_core = 0;
+       int cpu, best_cpu = -1, best_cpu_core = -1;
        struct cpumask *cpus;
+       bool on_idle_core;
 
        cpus = this_cpu_cpumask_var_ptr(select_rq_mask);
        cpumask_and(cpus, sched_domain_span(sd), p->cpus_ptr);

kernel/sched/fair.c select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int target) · line 7975

Quoted 7 lines
kernel/sched/fair.c +4 -0
@@ -7962,16 +7969,58 @@ select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int target)
                if (!choose_idle_cpu(cpu, p))
                        continue;
 
+               on_idle_core = is_core_idle(cpu);
+               if (prefer_idle_cores && !on_idle_core) {
+                       /* Track best among any idle CPU for fallback */
+                       fits = util_fits_cpu(task_util, util_min, util_max, cpu);

fits = util_fits_cpu(task_util, util_min, util_max, cpu); is always called so call it once above this if condition

this will help factorize the selection of best_cpu and best_cpu_core

Quoted 93 lines
kernel/sched/fair.c +57 -5
+                       if (fits > 0) {
+                               /*
+                                * Full fit: strictly better than fits 0 / -1;
+                                * among several, prefer higher capacity.
+                                */
+                               if (best_cpu < 0 || best_fits <= 0 ||
+                                   (best_fits > 0 && cpu_cap > best_cap)) {
+                                       best_cap = cpu_cap;
+                                       best_cpu = cpu;
+                                       best_fits = fits;
+                               }
+                               continue;
+                       }
+                       if (best_fits > 0)
+                               continue;
+                       if (fits < 0)
+                               cpu_cap = get_actual_cpu_capacity(cpu);
+                       if ((fits < best_fits) ||
+                           ((fits == best_fits) && (cpu_cap > best_cap))) {
+                               best_cap = cpu_cap;
+                               best_cpu = cpu;
+                               best_fits = fits;
+                       }
+                       continue;
+               }
+
                fits = util_fits_cpu(task_util, util_min, util_max, cpu);
 
                /* This CPU fits with all requirements */
-               if (fits > 0)
-                       return cpu;
+               if (fits > 0) {
+                       if (prefer_idle_cores && on_idle_core)
+                               return cpu;
+                       if (!prefer_idle_cores)
+                               return cpu;
+                       /*
+                        * Prefer idle cores: record and keep looking for
+                        * idle-core fit.
+                        */
+                       best_cap = cpu_cap;
+                       best_cpu = cpu;
+                       best_fits = fits;
+                       continue;
+               }
                /*
                 * Only the min performance hint (i.e. uclamp_min) doesn't fit.
                 * Look for the CPU with best capacity.
                 */
-               else if (fits < 0)
+               if (fits < 0)
                        cpu_cap = get_actual_cpu_capacity(cpu);
 
                /*
@@ -7984,8 +8033,17 @@ select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int target)
                        best_cpu = cpu;
                        best_fits = fits;
                }
+               if (prefer_idle_cores && on_idle_core &&
+                   ((fits < best_fits_core) ||
+                    ((fits == best_fits_core) && (cpu_cap > best_cap_core)))) {
+                       best_cap_core = cpu_cap;
+                       best_cpu_core = cpu;
+                       best_fits_core = fits;
+               }
        }
 
+       if (prefer_idle_cores && best_cpu_core >= 0)
+               return best_cpu_core;
        return best_cpu;
 }
 
@@ -7994,12 +8052,17 @@ static inline bool asym_fits_cpu(unsigned long util,
                                 unsigned long util_max,
                                 int cpu)
 {
-       if (sched_asym_cpucap_active())
+       if (sched_asym_cpucap_active()) {
                /*
                 * Return true only if the cpu fully fits the task requirements
                 * which include the utilization and the performance hints.
+                *
+                * When SMT is active, also require that the core has no busy
+                * siblings.
                 */
-               return (util_fits_cpu(util, util_min, util_max, cpu) > 0);
+               return (!sched_smt_active() || is_core_idle(cpu)) &&
+                      (util_fits_cpu(util, util_min, util_max, cpu) > 0);
+       }
 
        return true;
 }

kernel/sched/fair.c static int select_idle_sibling(struct task_struct *p, int prev, int target) · line 8164

Quoted 7 lines
kernel/sched/fair.c +2 -2
@@ -8097,8 +8160,9 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
                 * capacity path.
                 */
                if (sd) {
-                       i = select_idle_capacity(p, sd, target);
-                       return ((unsigned)i < nr_cpumask_bits) ? i : target;
+                       i = select_idle_capacity(p, sd, target,
+                               sched_smt_active() && test_idle_cores(target));

Move "sched_smt_active() && test_idle_cores(target)" inside select_idle_capacity(). I don't see the benefit of making it a parameter
or use has_idle_core for the parameter like other smt related function

Quoted 7 lines
kernel/sched/fair.c +1 -1
+                       return ((unsigned int)i < nr_cpumask_bits) ? i : target;
                }
        }
 
--

2.53.0

#3 lore raw

Re: [PATCH 1/4] sched/fair: Prefer fully-idle SMT cores in asym-capacity idle selection

Hi Vincent,

Quoted 79 lines · Vincent Guittot
Quoted 67 lines · Andrea Righi

On systems with asymmetric CPU capacity (e.g., ACPI/CPPC reporting different per-core frequencies), the wakeup path uses select_idle_capacity() and prioritizes idle CPUs with higher capacity for better task placement. However, when those CPUs belong to SMT cores, their effective capacity can be much lower than the nominal capacity when the sibling thread is busy: SMT siblings compete for shared resources, so a "high capacity" CPU that is idle but whose sibling is busy does not deliver its full capacity. This effective capacity reduction cannot be modeled by the static capacity value alone.

Introduce SMT awareness in the asym-capacity idle selection policy: when SMT is active prefer fully-idle SMT cores over partially-idle ones. A two-phase selection first tries only CPUs on fully idle cores, then falls back to any idle CPU if none fit.

Prioritizing fully-idle SMT cores yields better task placement because the effective capacity of partially-idle SMT cores is reduced; always preferring them when available leads to more accurate capacity usage on task wakeup.

On an SMT system with asymmetric CPU capacities, SMT-aware idle selection has been shown to improve throughput by around 15-18% for CPU-bound workloads, running an amount of tasks equal to the amount of SMT cores.

Cc Vincent Guittot <[email protected]>
Cc Dietmar Eggemann <[email protected]>
Cc Christian Loehle <[email protected]>
Cc Koba Ko <[email protected]>
---
 kernel/sched/fair.c | 86 +++++++++++++++++++++++++++++++++++++++------
 1 file changed, 75 insertions(+), 11 deletions(-)
kernel/sched/fair.c +11 -4
index d57c02e82f3a1..9a95628669851 100644
@@ -7940,14 +7940,21 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, bool
  * Scan the asym_capacity domain for idle CPUs; pick the first idle one on which
  * the task fits. If no CPU is big enough, but there are idle ones, try to
  * maximize capacity.
+ *
+ * When @prefer_idle_cores is true (asym + SMT and idle cores exist), prefer
+ * CPUs on fully-idle cores over partially-idle ones in a single pass: track
+ * the best candidate among idle-core CPUs and the best among any idle CPU,
+ * then return the idle-core candidate if found, else the best any-idle.
  */
 static int
-select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int target)
+select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int target,
+                    bool prefer_idle_cores)
 {
-       unsigned long task_util, util_min, util_max, best_cap = 0;
-       int fits, best_fits = 0;
-       int cpu, best_cpu = -1;
+       unsigned long task_util, util_min, util_max, best_cap = 0, best_cap_core = 0;
+       int fits, best_fits = 0, best_fits_core = 0;
+       int cpu, best_cpu = -1, best_cpu_core = -1;
        struct cpumask *cpus;
+       bool on_idle_core;
 
        cpus = this_cpu_cpumask_var_ptr(select_rq_mask);
        cpumask_and(cpus, sched_domain_span(sd), p->cpus_ptr);

kernel/sched/fair.c select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int target) · line 7975

Quoted 7 lines
kernel/sched/fair.c +4 -0
@@ -7962,16 +7969,58 @@ select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int target)
                if (!choose_idle_cpu(cpu, p))
                        continue;
 
+               on_idle_core = is_core_idle(cpu);
+               if (prefer_idle_cores && !on_idle_core) {
+                       /* Track best among any idle CPU for fallback */
+                       fits = util_fits_cpu(task_util, util_min, util_max, cpu);

fits = util_fits_cpu(task_util, util_min, util_max, cpu); is always called so call it once above this if condition

Quoted 1 line

this will help factorize the selection of best_cpu and best_cpu_core

Makes sense.

Quoted 91 lines
Quoted 91 lines
kernel/sched/fair.c +57 -5
+                       if (fits > 0) {
+                               /*
+                                * Full fit: strictly better than fits 0 / -1;
+                                * among several, prefer higher capacity.
+                                */
+                               if (best_cpu < 0 || best_fits <= 0 ||
+                                   (best_fits > 0 && cpu_cap > best_cap)) {
+                                       best_cap = cpu_cap;
+                                       best_cpu = cpu;
+                                       best_fits = fits;
+                               }
+                               continue;
+                       }
+                       if (best_fits > 0)
+                               continue;
+                       if (fits < 0)
+                               cpu_cap = get_actual_cpu_capacity(cpu);
+                       if ((fits < best_fits) ||
+                           ((fits == best_fits) && (cpu_cap > best_cap))) {
+                               best_cap = cpu_cap;
+                               best_cpu = cpu;
+                               best_fits = fits;
+                       }
+                       continue;
+               }
+
                fits = util_fits_cpu(task_util, util_min, util_max, cpu);
 
                /* This CPU fits with all requirements */
-               if (fits > 0)
-                       return cpu;
+               if (fits > 0) {
+                       if (prefer_idle_cores && on_idle_core)
+                               return cpu;
+                       if (!prefer_idle_cores)
+                               return cpu;
+                       /*
+                        * Prefer idle cores: record and keep looking for
+                        * idle-core fit.
+                        */
+                       best_cap = cpu_cap;
+                       best_cpu = cpu;
+                       best_fits = fits;
+                       continue;
+               }
                /*
                 * Only the min performance hint (i.e. uclamp_min) doesn't fit.
                 * Look for the CPU with best capacity.
                 */
-               else if (fits < 0)
+               if (fits < 0)
                        cpu_cap = get_actual_cpu_capacity(cpu);
 
                /*
@@ -7984,8 +8033,17 @@ select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int target)
                        best_cpu = cpu;
                        best_fits = fits;
                }
+               if (prefer_idle_cores && on_idle_core &&
+                   ((fits < best_fits_core) ||
+                    ((fits == best_fits_core) && (cpu_cap > best_cap_core)))) {
+                       best_cap_core = cpu_cap;
+                       best_cpu_core = cpu;
+                       best_fits_core = fits;
+               }
        }
 
+       if (prefer_idle_cores && best_cpu_core >= 0)
+               return best_cpu_core;
        return best_cpu;
 }
 
@@ -7994,12 +8052,17 @@ static inline bool asym_fits_cpu(unsigned long util,
                                 unsigned long util_max,
                                 int cpu)
 {
-       if (sched_asym_cpucap_active())
+       if (sched_asym_cpucap_active()) {
                /*
                 * Return true only if the cpu fully fits the task requirements
                 * which include the utilization and the performance hints.
+                *
+                * When SMT is active, also require that the core has no busy
+                * siblings.
                 */
-               return (util_fits_cpu(util, util_min, util_max, cpu) > 0);
+               return (!sched_smt_active() || is_core_idle(cpu)) &&
+                      (util_fits_cpu(util, util_min, util_max, cpu) > 0);
+       }
 
Quoted 15 lines

kernel/sched/fair.c static int select_idle_sibling(struct task_struct *p, int prev, int target) · line 8164

Quoted 10 lines
kernel/sched/fair.c +2 -2
        return true;
 }
@@ -8097,8 +8160,9 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
                 * capacity path.
                 */
                if (sd) {
-                       i = select_idle_capacity(p, sd, target);
-                       return ((unsigned)i < nr_cpumask_bits) ? i : target;
+                       i = select_idle_capacity(p, sd, target,
+                               sched_smt_active() && test_idle_cores(target));

Move "sched_smt_active() && test_idle_cores(target)" inside select_idle_capacity(). I don't see the benefit of making it a parameter
or use has_idle_core for the parameter like other smt related function

And also makes sense.

Quoted 9 lines
Quoted 9 lines
kernel/sched/fair.c +1 -1
+                       return ((unsigned int)i < nr_cpumask_bits) ? i : target;
                }
        }
 
--

2.53.0

Thanks,
-Andrea

#4 lore raw

Re: [PATCH 1/4] sched/fair: Prefer fully-idle SMT cores in asym-capacity idle selection

In reply to Andrea Righi

Hello Andrea,

kernel/sched/fair.c select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int target) · line 8009

Quoted 8 lines · Andrea Righi
kernel/sched/fair.c +5 -2
 		/* This CPU fits with all requirements */
-		if (fits > 0)
-			return cpu;
+		if (fits > 0) {
+			if (prefer_idle_cores && on_idle_core)
+				return cpu;
+			if (!prefer_idle_cores)
+				return cpu;

nit.

Can the above two be re-wittern as:

    if (!prefer_idle_cores || on_idle_core)
        return cpu; 

since they are equivalent.

Signature · 2 lines
Thanks and Regards,
Prateek
#5 lore raw

Re: [PATCH 1/4] sched/fair: Prefer fully-idle SMT cores in asym-capacity idle selection

Hi Prateek,

Quoted 14 lines · K Prateek Nayak

Hello Andrea,

kernel/sched/fair.c select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int target) · line 8009

Quoted 8 lines · Andrea Righi
kernel/sched/fair.c +5 -2
 		/* This CPU fits with all requirements */
-		if (fits > 0)
-			return cpu;
+		if (fits > 0) {
+			if (prefer_idle_cores && on_idle_core)
+				return cpu;
+			if (!prefer_idle_cores)
+				return cpu;

nit.

Quoted 6 lines

Can the above two be re-wittern as:

    if (!prefer_idle_cores || on_idle_core)
        return cpu; 

since they are equivalent.

Oh yes, indeed.

Thanks,
-Andrea

#6 lore raw

Re: [PATCH 1/4] sched/fair: Prefer fully-idle SMT cores in asym-capacity idle selection

Hello Andrea,

Quoted 15 lines · Andrea Righi
Quoted 14 lines · K Prateek Nayak

Hello Andrea,

Review

Quoted 8 lines · Andrea Righi
            /* This CPU fits with all requirements */
(quoted diff) +5 -2
-           if (fits > 0)
-                   return cpu;
+           if (fits > 0) {
+                   if (prefer_idle_cores && on_idle_core)
+                           return cpu;
+                   if (!prefer_idle_cores)
+                           return cpu;

nit.

Quoted 8 lines
Quoted 6 lines

Can the above two be re-wittern as:

    if (!prefer_idle_cores || on_idle_core)
        return cpu;

since they are equivalent.

Oh yes, indeed.

Also, can we just rewrite this Patch as:

(Includes feedback from Vincent; Only build tested)

kernel/sched/fair.c +11 -1
index 700d0f145ca6..cffd5649b54e 100644
@@ -7946,6 +7946,7 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, bool
 static int
 select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int target)
 {
+	bool prefers_idle_core = sched_smt_active() && test_idle_cores(target);
 	unsigned long task_util, util_min, util_max, best_cap = 0;
 	int fits, best_fits = 0;
 	int cpu, best_cpu = -1;
@@ -7959,6 +7960,7 @@ select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int target)
 	util_max = uclamp_eff_value(p, UCLAMP_MAX);
 
 	for_each_cpu_wrap(cpu, cpus, target) {
+		bool preferred_core = !prefers_idle_core || is_core_idle(cpu);
 		unsigned long cpu_cap = capacity_of(cpu);
 
 		if (!choose_idle_cpu(cpu, p))
@@ -7967,7 +7969,7 @@ select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int target)
 		fits = util_fits_cpu(task_util, util_min, util_max, cpu);
 
 		/* This CPU fits with all requirements */
-		if (fits > 0)
+		if (fits > 0 && preferred_core)
 			return cpu;
 		/*
 		 * Only the min performance hint (i.e. uclamp_min) doesn't fit.
@@ -7976,6 +7978,14 @@ select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int target)
 		else if (fits < 0)
 			cpu_cap = get_actual_cpu_capacity(cpu);
 
+		/*
+		 * If we are on an preferred core, translate the range of fits
+		 * from [-1, 1] to [-4, -2]. This ensures that an idle core
+		 * is always given priority over (paritally) busy core.
+		 */
+		if (preferred_core)
+			fits -= 3;
+
 		/*
 		 * First, select CPU which fits better (-1 being better than 0).
 		 * Then, select the one with best capacity at same level.

---

My naive eyes say it should be equivalent of what you have but maybe I'm wrong?

Signature · 2 lines
Thanks and Regards,
Prateek
#7 lore raw

Re: [PATCH 1/4] sched/fair: Prefer fully-idle SMT cores in asym-capacity idle selection

Hi Prateek,

Quoted 62 lines · K Prateek Nayak

Hello Andrea,

Quoted 15 lines · Andrea Righi
Quoted 14 lines · K Prateek Nayak

Hello Andrea,

Review

Quoted 8 lines · Andrea Righi
            /* This CPU fits with all requirements */
(quoted diff) +5 -2
-           if (fits > 0)
-                   return cpu;
+           if (fits > 0) {
+                   if (prefer_idle_cores && on_idle_core)
+                           return cpu;
+                   if (!prefer_idle_cores)
+                           return cpu;

nit.

Quoted 8 lines
Quoted 6 lines

Can the above two be re-wittern as:

    if (!prefer_idle_cores || on_idle_core)
        return cpu;

since they are equivalent.

Oh yes, indeed.

Also, can we just rewrite this Patch as:

(Includes feedback from Vincent; Only build tested)

kernel/sched/fair.c +3 -1
index 700d0f145ca6..cffd5649b54e 100644
@@ -7946,6 +7946,7 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, bool
 static int
 select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int target)
 {
+	bool prefers_idle_core = sched_smt_active() && test_idle_cores(target);
 	unsigned long task_util, util_min, util_max, best_cap = 0;
 	int fits, best_fits = 0;
 	int cpu, best_cpu = -1;
@@ -7959,6 +7960,7 @@ select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int target)
 	util_max = uclamp_eff_value(p, UCLAMP_MAX);
 
 	for_each_cpu_wrap(cpu, cpus, target) {
+		bool preferred_core = !prefers_idle_core || is_core_idle(cpu);
 		unsigned long cpu_cap = capacity_of(cpu);
 
 		if (!choose_idle_cpu(cpu, p))
@@ -7967,7 +7969,7 @@ select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int target)
 		fits = util_fits_cpu(task_util, util_min, util_max, cpu);
 
 		/* This CPU fits with all requirements */
-		if (fits > 0)
+		if (fits > 0 && preferred_core)
 			return cpu;
 		/*
 		 * Only the min performance hint (i.e. uclamp_min) doesn't fit.
@@ -7976,6 +7978,14 @@ select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int target)
 		else if (fits < 0)

kernel/sched/fair.c select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int target) · line 7988

Quoted 10 lines
kernel/sched/fair.c +8 -0
 			cpu_cap = get_actual_cpu_capacity(cpu);
 
+		/*
+		 * If we are on an preferred core, translate the range of fits
+		 * from [-1, 1] to [-4, -2]. This ensures that an idle core
+		 * is always given priority over (paritally) busy core.
+		 */
+		if (preferred_core)
+			fits -= 3;
+

Ah, I like this trick. Yes, this definitely makes the patch more compact.

Quoted 7 lines
kernel/sched/fair.c +0 -1
 		/*
 		 * First, select CPU which fits better (-1 being better than 0).
 		 * Then, select the one with best capacity at same level.
---

My naive eyes say it should be equivalent of what you have but maybe I'm wrong?

It seems correct to my naive eyes as well. Will test this out to make sure.

Unfortunately I just lost access to my system (bummer), I found another Vera machine, but this one has a version of the firmware that exposes all CPUs with the same highest_perf... so I can still do some testing, but not the same one with SD_ASYM_CPUCAPACITY + SMT. I should get access to the previous system with the different highest_perf values on Monday.

Thanks,
-Andrea

#8 lore raw

Re: [PATCH 1/4] sched/fair: Prefer fully-idle SMT cores in asym-capacity idle selection

Hello Andrea,

Quoted 4 lines · Andrea Righi
Quoted 2 lines

My naive eyes say it should be equivalent of what you have but maybe I'm wrong?

It seems correct to my naive eyes as well. Will test this out to make sure.

So I found one small problem with fits > 0 && !preferred_core where even though it is an ideal target, we don't end up preferring it because of the larger "fits" value.

Here is an updated diff:

(Only build tested)

kernel/sched/fair.c +25 -2
index 226509231e67..580218656865 100644
@@ -7949,6 +7949,7 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, bool
 static int
 select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int target)
 {
+	bool prefers_idle_core = sched_smt_active() && test_idle_cores(target);
 	unsigned long task_util, util_min, util_max, best_cap = 0;
 	int fits, best_fits = 0;
 	int cpu, best_cpu = -1;
@@ -7962,6 +7963,7 @@ select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int target)
 	util_max = uclamp_eff_value(p, UCLAMP_MAX);
 
 	for_each_cpu_wrap(cpu, cpus, target) {
+		bool preferred_core = !prefers_idle_core || is_core_idle(cpu);
 		unsigned long cpu_cap = capacity_of(cpu);
 
 		if (!choose_idle_cpu(cpu, p))
@@ -7970,7 +7972,7 @@ select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int target)
 		fits = util_fits_cpu(task_util, util_min, util_max, cpu);
 
 		/* This CPU fits with all requirements */
-		if (fits > 0)
+		if (fits > 0 && preferred_core)
 			return cpu;
 		/*
 		 * Only the min performance hint (i.e. uclamp_min) doesn't fit.
@@ -7978,9 +7980,30 @@ select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int target)
 		 */
 		else if (fits < 0)
 			cpu_cap = get_actual_cpu_capacity(cpu);
+		/*
+		 * fits > 0 implies we are not on a preferred core
+		 * but the util fits CPU capacity. Set fits to -2 so
+		 * the effective range becomes [-2, 0] where:
+		 *    0 - does not fit
+		 *   -1 - fits with the exception of UCLAMP_MIN
+		 *   -2 - fits with the exception of preferred_core
+		 */
+		else if (fits > 0)
+			fits = -2;
+
+		/*
+		 * If we are on an preferred core, translate the range of fits
+		 * of [-1, 0] to [-4, -3]. This ensures that an idle core
+		 * is always given priority over (partially) busy core.
+		 *
+		 * A fully fitting idle core would have returned early and hence
+		 * fits > 0 for preferred_core need not be dealt with.
+		 */
+		if (preferred_core)
+			fits -= 3;
 
 		/*
-		 * First, select CPU which fits better (-1 being better than 0).
+		 * First, select CPU which fits better (lower is more preferred).
 		 * Then, select the one with best capacity at same level.
 		 */
 		if ((fits < best_fits) ||

---

Sorry for the oversight but this should now be equivalent to your Patch 1. I'll let Vincent comment if he prefers this to the original or not :-)

Signature · 2 lines
Thanks and Regards,
Prateek
maintainer
#9 lore raw

Re: [PATCH 1/4] sched/fair: Prefer fully-idle SMT cores in asym-capacity idle selection

Quoted 80 lines · K Prateek Nayak

Hello Andrea,

Quoted 4 lines · Andrea Righi
Quoted 2 lines

My naive eyes say it should be equivalent of what you have but maybe I'm wrong?

It seems correct to my naive eyes as well. Will test this out to make sure.

So I found one small problem with fits > 0 && !preferred_core where even though it is an ideal target, we don't end up preferring it because of the larger "fits" value.

Here is an updated diff:

(Only build tested)

kernel/sched/fair.c +25 -3
index 226509231e67..580218656865 100644
@@ -7949,6 +7949,7 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, bool
 static int
 select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int target)
 {
+       bool prefers_idle_core = sched_smt_active() && test_idle_cores(target);
        unsigned long task_util, util_min, util_max, best_cap = 0;
        int fits, best_fits = 0;
        int cpu, best_cpu = -1;
@@ -7962,6 +7963,7 @@ select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int target)
        util_max = uclamp_eff_value(p, UCLAMP_MAX);
 
        for_each_cpu_wrap(cpu, cpus, target) {
+               bool preferred_core = !prefers_idle_core || is_core_idle(cpu);
                unsigned long cpu_cap = capacity_of(cpu);
 
                if (!choose_idle_cpu(cpu, p))
@@ -7970,7 +7972,7 @@ select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int target)
                fits = util_fits_cpu(task_util, util_min, util_max, cpu);
 
                /* This CPU fits with all requirements */
-               if (fits > 0)
+               if (fits > 0 && preferred_core)
                        return cpu;
                /*
                 * Only the min performance hint (i.e. uclamp_min) doesn't fit.
@@ -7978,9 +7980,30 @@ select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int target)
                 */
                else if (fits < 0)
                        cpu_cap = get_actual_cpu_capacity(cpu);
+               /*
+                * fits > 0 implies we are not on a preferred core
+                * but the util fits CPU capacity. Set fits to -2 so
+                * the effective range becomes [-2, 0] where:
+                *    0 - does not fit
+                *   -1 - fits with the exception of UCLAMP_MIN
+                *   -2 - fits with the exception of preferred_core
+                */
+               else if (fits > 0)
+                       fits = -2;
+
+               /*
+                * If we are on an preferred core, translate the range of fits
+                * of [-1, 0] to [-4, -3]. This ensures that an idle core
+                * is always given priority over (partially) busy core.
+                *
+                * A fully fitting idle core would have returned early and hence
+                * fits > 0 for preferred_core need not be dealt with.
+                */
+               if (preferred_core)
+                       fits -= 3;
 
                /*
-                * First, select CPU which fits better (-1 being better than 0).
+                * First, select CPU which fits better (lower is more preferred).
                 * Then, select the one with best capacity at same level.
                 */
                if ((fits < best_fits) ||
---
Quoted 3 lines

Sorry for the oversight but this should now be equivalent to your Patch 1. I'll let Vincent comment if he prefers this to the original or not :-)

Yes, I prefer this version which keeps the same logic for selecting the best cpu

Thanks
Vincent

Quoted 5 lines

--
Thanks and Regards,
Prateek

#10 lore raw

Re: [PATCH 1/4] sched/fair: Prefer fully-idle SMT cores in asym-capacity idle selection

In reply to K Prateek Nayak

Hi Prateek,

Quoted 8 lines · K Prateek Nayak

Hello Andrea,

Quoted 4 lines · Andrea Righi
Quoted 2 lines

My naive eyes say it should be equivalent of what you have but maybe I'm wrong?

It seems correct to my naive eyes as well. Will test this out to make sure.

Quoted 7 lines

So I found one small problem with fits > 0 && !preferred_core where even though it is an ideal target, we don't end up preferring it because of the larger "fits" value.

Here is an updated diff:

(Only build tested)

I'm getting worse performance with this one (but better than mainline). I'm trying to understand why.

BTW, we also need to fix asym_fits_cpu() to do something like this:

	return (!sched_smt_active() || is_core_idle(cpu)) &&
	       (util_fits_cpu(util, util_min, util_max, cpu) > 0);

...or we'd return early from select_idle_sibling() with busy SMT cores.

Thanks,
-Andrea

Quoted 72 lines
kernel/sched/fair.c +25 -3
index 226509231e67..580218656865 100644
@@ -7949,6 +7949,7 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, bool
 static int
 select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int target)
 {
+	bool prefers_idle_core = sched_smt_active() && test_idle_cores(target);
 	unsigned long task_util, util_min, util_max, best_cap = 0;
 	int fits, best_fits = 0;
 	int cpu, best_cpu = -1;
@@ -7962,6 +7963,7 @@ select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int target)
 	util_max = uclamp_eff_value(p, UCLAMP_MAX);
 
 	for_each_cpu_wrap(cpu, cpus, target) {
+		bool preferred_core = !prefers_idle_core || is_core_idle(cpu);
 		unsigned long cpu_cap = capacity_of(cpu);
 
 		if (!choose_idle_cpu(cpu, p))
@@ -7970,7 +7972,7 @@ select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int target)
 		fits = util_fits_cpu(task_util, util_min, util_max, cpu);
 
 		/* This CPU fits with all requirements */
-		if (fits > 0)
+		if (fits > 0 && preferred_core)
 			return cpu;
 		/*
 		 * Only the min performance hint (i.e. uclamp_min) doesn't fit.
@@ -7978,9 +7980,30 @@ select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int target)
 		 */
 		else if (fits < 0)
 			cpu_cap = get_actual_cpu_capacity(cpu);
+		/*
+		 * fits > 0 implies we are not on a preferred core
+		 * but the util fits CPU capacity. Set fits to -2 so
+		 * the effective range becomes [-2, 0] where:
+		 *    0 - does not fit
+		 *   -1 - fits with the exception of UCLAMP_MIN
+		 *   -2 - fits with the exception of preferred_core
+		 */
+		else if (fits > 0)
+			fits = -2;
+
+		/*
+		 * If we are on an preferred core, translate the range of fits
+		 * of [-1, 0] to [-4, -3]. This ensures that an idle core
+		 * is always given priority over (partially) busy core.
+		 *
+		 * A fully fitting idle core would have returned early and hence
+		 * fits > 0 for preferred_core need not be dealt with.
+		 */
+		if (preferred_core)
+			fits -= 3;
 
 		/*
-		 * First, select CPU which fits better (-1 being better than 0).
+		 * First, select CPU which fits better (lower is more preferred).
 		 * Then, select the one with best capacity at same level.
 		 */
 		if ((fits < best_fits) ||
---

Sorry for the oversight but this should now be equivalent to your Patch 1. I'll let Vincent comment if he prefers this to the original or not :-)

Signature · 2 lines
Thanks and Regards,
Prateek
#11 lore raw

Re: [PATCH 1/4] sched/fair: Prefer fully-idle SMT cores in asym-capacity idle selection

Quoted 15 lines · Andrea Righi

Hi Prateek,

Quoted 12 lines · K Prateek Nayak

Hello Andrea,

Quoted 4 lines · Andrea Righi
Quoted 2 lines

My naive eyes say it should be equivalent of what you have but maybe I'm wrong?

It seems correct to my naive eyes as well. Will test this out to make sure.

So I found one small problem with fits > 0 && !preferred_core where even though it is an ideal target, we don't end up preferring it because of the larger "fits" value.

Quoted 6 lines
Quoted 3 lines

Here is an updated diff:

(Only build tested)

I'm getting worse performance with this one (but better than mainline). I'm trying to understand why.

Nevermind...

Quoted 7 lines

BTW, we also need to fix asym_fits_cpu() to do something like this:

	return (!sched_smt_active() || is_core_idle(cpu)) &&
	       (util_fits_cpu(util, util_min, util_max, cpu) > 0);

...or we'd return early from select_idle_sibling() with busy SMT cores.

...I was actually missing this piece right here. So, everything looks good with this extra change applied.

I'll repeat all my tests just in case and will send a new version with your changes.

Thanks!
-Andrea

#12 lore raw

PATCH 2/4 sched/fair: Reject misfit pulls onto busy SMT siblings on asym-capacity

In reply to Andrea Righi

When SD_ASYM_CPUCAPACITY load balancing considers pulling a misfit task, capacity_of(dst_cpu) can overstate available compute if the SMT sibling is
busy: the core does not deliver its full nominal capacity.

If SMT is active and dst_cpu is not on a fully idle core, skip this destination so we do not migrate a misfit expecting a capacity upgrade we cannot actually provide.

Cc Vincent Guittot <[email protected]>
Cc Dietmar Eggemann <[email protected]>
Cc Christian Loehle <[email protected]>
Cc Koba Ko <[email protected]>
---
 kernel/sched/fair.c | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)
kernel/sched/fair.c +7 -1
index 9a95628669851..f8deaaa5bfc85 100644
@@ -10819,10 +10819,16 @@ static bool update_sd_pick_busiest(struct lb_env *env,
 	 * We can use max_capacity here as reduction in capacity on some
 	 * CPUs in the group should either be possible to resolve
 	 * internally or be covered by avg_load imbalance (eventually).
+	 *
+	 * When SMT is active, only pull a misfit to dst_cpu if it is on a
+	 * fully idle core; otherwise the effective capacity of the core is
+	 * reduced and we may not actually provide more capacity than the
+	 * source.
 	 */
 	if ((env->sd->flags & SD_ASYM_CPUCAPACITY) &&
 	    (sgs->group_type == group_misfit_task) &&
-	    (!capacity_greater(capacity_of(env->dst_cpu), sg->sgc->max_capacity) ||
+	    ((sched_smt_active() && !is_core_idle(env->dst_cpu)) ||
+	     !capacity_greater(capacity_of(env->dst_cpu), sg->sgc->max_capacity) ||
 	     sds->local_stat.group_type != group_has_spare))
 		return false;
 
Signature · 1 line
2.53.0
#13 lore raw

PATCH 3/4 sched/fair: Enable EAS with SMT on SD_ASYM_CPUCAPACITY systems

In reply to Andrea Righi

Drop the sched_is_eas_possible() guard that rejects EAS whenever SMT is active. This allows to enable EAS and perf-domain setup to succeed on SD_ASYM_CPUCAPACITY topologies with SMT enabled.

Moreover, apply to find_energy_efficient_cpu() the same SMT-aware preference as the non-EAS wakeup path: when SMT is active and there is a fully-idle core in the relevant domain, prefer max-spare-capacity candidates on fully-idle cores. Otherwise, fall back to the prior behavior, to include also partially-idle SMT siblings.

Cc Vincent Guittot <[email protected]>
Cc Dietmar Eggemann <[email protected]>
Cc Christian Loehle <[email protected]>
Cc Koba Ko <[email protected]>
---
 kernel/sched/fair.c     | 50 +++++++++++++++++++++++++++++++++++++++--
 kernel/sched/topology.c |  9 --------
 2 files changed, 48 insertions(+), 11 deletions(-)
kernel/sched/fair.c +48 -2
index f8deaaa5bfc85..593a89f688679 100644
@@ -8658,13 +8658,15 @@ static int find_energy_efficient_cpu(struct task_struct *p, int prev_cpu)
 	eenv_task_busy_time(&eenv, p, prev_cpu);
 
 	for (; pd; pd = pd->next) {
-		unsigned long util_min = p_util_min, util_max = p_util_max;
 		unsigned long cpu_cap, cpu_actual_cap, util;
 		long prev_spare_cap = -1, max_spare_cap = -1;
+		long max_spare_cap_fallback = -1;
 		unsigned long rq_util_min, rq_util_max;
 		unsigned long cur_delta, base_energy;
-		int max_spare_cap_cpu = -1;
+		int max_spare_cap_cpu = -1, max_spare_cap_cpu_fallback = -1;
 		int fits, max_fits = -1;
+		int max_fits_fallback = -1;
+		bool prefer_idle_cores;
 
 		if (!cpumask_and(cpus, perf_domain_span(pd), cpu_online_mask))
 			continue;
@@ -8676,6 +8678,8 @@ static int find_energy_efficient_cpu(struct task_struct *p, int prev_cpu)
 		eenv.cpu_cap = cpu_actual_cap;
 		eenv.pd_cap = 0;
 
+		prefer_idle_cores = sched_smt_active() && test_idle_cores(prev_cpu);
+
 		for_each_cpu(cpu, cpus) {
 			struct rq *rq = cpu_rq(cpu);
 
@@ -8687,6 +8691,11 @@ static int find_energy_efficient_cpu(struct task_struct *p, int prev_cpu)
 			if (!cpumask_test_cpu(cpu, p->cpus_ptr))
 				continue;
 
+			if (prefer_idle_cores && cpu != prev_cpu && !is_core_idle(cpu))
+				goto fallback;
+
+			unsigned long util_min = p_util_min, util_max = p_util_max;
+
 			util = cpu_util(cpu, p, cpu, 0);
 			cpu_cap = capacity_of(cpu);
 
@@ -8733,6 +8742,43 @@ static int find_energy_efficient_cpu(struct task_struct *p, int prev_cpu)
 				max_spare_cap_cpu = cpu;
 				max_fits = fits;
 			}
+
+fallback:
+			if (!prefer_idle_cores || cpu == prev_cpu || is_core_idle(cpu))
+				continue;
+
+			util_min = p_util_min;
+			util_max = p_util_max;
+			util = cpu_util(cpu, p, cpu, 0);
+			cpu_cap = capacity_of(cpu);
+
+			if (uclamp_is_used() && !uclamp_rq_is_idle(rq)) {
+				rq_util_min = uclamp_rq_get(rq, UCLAMP_MIN);
+				rq_util_max = uclamp_rq_get(rq, UCLAMP_MAX);
+
+				util_min = max(rq_util_min, p_util_min);
+				util_max = max(rq_util_max, p_util_max);
+			}
+
+			fits = util_fits_cpu(util, util_min, util_max, cpu);
+			if (!fits)
+				continue;
+
+			lsub_positive(&cpu_cap, util);
+
+			if ((fits > max_fits_fallback) ||
+			    ((fits == max_fits_fallback) &&
+			     ((long)cpu_cap > max_spare_cap_fallback))) {
+				max_spare_cap_fallback = cpu_cap;
+				max_spare_cap_cpu_fallback = cpu;
+				max_fits_fallback = fits;
+			}
+		}
+
+		if (max_spare_cap_cpu < 0 && max_spare_cap_cpu_fallback >= 0) {
+			max_spare_cap = max_spare_cap_fallback;
+			max_spare_cap_cpu = max_spare_cap_cpu_fallback;
+			max_fits = max_fits_fallback;
 		}
 
 		if (max_spare_cap_cpu < 0 && prev_spare_cap < 0)
kernel/sched/topology.c +0 -9
index 061f8c85f5552..cb060fe56aec1 100644
@@ -232,15 +232,6 @@ static bool sched_is_eas_possible(const struct cpumask *cpu_mask)
 		return false;
 	}
 
-	/* EAS definitely does *not* handle SMT */
-	if (sched_smt_active()) {
-		if (sched_debug()) {
-			pr_info("rd %*pbl: Checking EAS, SMT is not supported\n",
-				cpumask_pr_args(cpu_mask));
-		}
-		return false;
-	}
-
 	if (!arch_scale_freq_invariant()) {
 		if (sched_debug()) {
 			pr_info("rd %*pbl: Checking EAS: frequency-invariant load tracking not yet supported",
Signature · 1 line
2.53.0
maintainer
#14 lore raw

Re: [PATCH 3/4] sched/fair: Enable EAS with SMT on SD_ASYM_CPUCAPACITY systems

Quoted 4 lines · Andrea Righi

Drop the sched_is_eas_possible() guard that rejects EAS whenever SMT is active. This allows to enable EAS and perf-domain setup to succeed on SD_ASYM_CPUCAPACITY topologies with SMT enabled.

I don't think that we want to enable EAS with SMT. So keep EAS and SMT exclusive, at least for now

Quoted 128 lines

Moreover, apply to find_energy_efficient_cpu() the same SMT-aware preference as the non-EAS wakeup path: when SMT is active and there is a fully-idle core in the relevant domain, prefer max-spare-capacity candidates on fully-idle cores. Otherwise, fall back to the prior behavior, to include also partially-idle SMT siblings.

Cc Vincent Guittot <[email protected]>
Cc Dietmar Eggemann <[email protected]>
Cc Christian Loehle <[email protected]>
Cc Koba Ko <[email protected]>
---
 kernel/sched/fair.c     | 50 +++++++++++++++++++++++++++++++++++++++--
 kernel/sched/topology.c |  9 --------
 2 files changed, 48 insertions(+), 11 deletions(-)
kernel/sched/fair.c +48 -2
index f8deaaa5bfc85..593a89f688679 100644
@@ -8658,13 +8658,15 @@ static int find_energy_efficient_cpu(struct task_struct *p, int prev_cpu)
        eenv_task_busy_time(&eenv, p, prev_cpu);
 
        for (; pd; pd = pd->next) {
-               unsigned long util_min = p_util_min, util_max = p_util_max;
                unsigned long cpu_cap, cpu_actual_cap, util;
                long prev_spare_cap = -1, max_spare_cap = -1;
+               long max_spare_cap_fallback = -1;
                unsigned long rq_util_min, rq_util_max;
                unsigned long cur_delta, base_energy;
-               int max_spare_cap_cpu = -1;
+               int max_spare_cap_cpu = -1, max_spare_cap_cpu_fallback = -1;
                int fits, max_fits = -1;
+               int max_fits_fallback = -1;
+               bool prefer_idle_cores;
 
                if (!cpumask_and(cpus, perf_domain_span(pd), cpu_online_mask))
                        continue;
@@ -8676,6 +8678,8 @@ static int find_energy_efficient_cpu(struct task_struct *p, int prev_cpu)
                eenv.cpu_cap = cpu_actual_cap;
                eenv.pd_cap = 0;
 
+               prefer_idle_cores = sched_smt_active() && test_idle_cores(prev_cpu);
+
                for_each_cpu(cpu, cpus) {
                        struct rq *rq = cpu_rq(cpu);
 
@@ -8687,6 +8691,11 @@ static int find_energy_efficient_cpu(struct task_struct *p, int prev_cpu)
                        if (!cpumask_test_cpu(cpu, p->cpus_ptr))
                                continue;
 
+                       if (prefer_idle_cores && cpu != prev_cpu && !is_core_idle(cpu))
+                               goto fallback;
+
+                       unsigned long util_min = p_util_min, util_max = p_util_max;
+
                        util = cpu_util(cpu, p, cpu, 0);
                        cpu_cap = capacity_of(cpu);
 
@@ -8733,6 +8742,43 @@ static int find_energy_efficient_cpu(struct task_struct *p, int prev_cpu)
                                max_spare_cap_cpu = cpu;
                                max_fits = fits;
                        }
+
+fallback:
+                       if (!prefer_idle_cores || cpu == prev_cpu || is_core_idle(cpu))
+                               continue;
+
+                       util_min = p_util_min;
+                       util_max = p_util_max;
+                       util = cpu_util(cpu, p, cpu, 0);
+                       cpu_cap = capacity_of(cpu);
+
+                       if (uclamp_is_used() && !uclamp_rq_is_idle(rq)) {
+                               rq_util_min = uclamp_rq_get(rq, UCLAMP_MIN);
+                               rq_util_max = uclamp_rq_get(rq, UCLAMP_MAX);
+
+                               util_min = max(rq_util_min, p_util_min);
+                               util_max = max(rq_util_max, p_util_max);
+                       }
+
+                       fits = util_fits_cpu(util, util_min, util_max, cpu);
+                       if (!fits)
+                               continue;
+
+                       lsub_positive(&cpu_cap, util);
+
+                       if ((fits > max_fits_fallback) ||
+                           ((fits == max_fits_fallback) &&
+                            ((long)cpu_cap > max_spare_cap_fallback))) {
+                               max_spare_cap_fallback = cpu_cap;
+                               max_spare_cap_cpu_fallback = cpu;
+                               max_fits_fallback = fits;
+                       }
+               }
+
+               if (max_spare_cap_cpu < 0 && max_spare_cap_cpu_fallback >= 0) {
+                       max_spare_cap = max_spare_cap_fallback;
+                       max_spare_cap_cpu = max_spare_cap_cpu_fallback;
+                       max_fits = max_fits_fallback;
                }
 
                if (max_spare_cap_cpu < 0 && prev_spare_cap < 0)
kernel/sched/topology.c +0 -10
index 061f8c85f5552..cb060fe56aec1 100644
@@ -232,15 +232,6 @@ static bool sched_is_eas_possible(const struct cpumask *cpu_mask)
                return false;
        }
 
-       /* EAS definitely does *not* handle SMT */
-       if (sched_smt_active()) {
-               if (sched_debug()) {
-                       pr_info("rd %*pbl: Checking EAS, SMT is not supported\n",
-                               cpumask_pr_args(cpu_mask));
-               }
-               return false;
-       }
-
        if (!arch_scale_freq_invariant()) {
                if (sched_debug()) {
                        pr_info("rd %*pbl: Checking EAS: frequency-invariant load tracking not yet supported",
--

2.53.0

#15 lore raw

Re: [PATCH 3/4] sched/fair: Enable EAS with SMT on SD_ASYM_CPUCAPACITY systems

Quoted 8 lines · Vincent Guittot
Quoted 4 lines · Andrea Righi

Drop the sched_is_eas_possible() guard that rejects EAS whenever SMT is active. This allows to enable EAS and perf-domain setup to succeed on SD_ASYM_CPUCAPACITY topologies with SMT enabled.

I don't think that we want to enable EAS with SMT. So keep EAS and SMT exclusive, at least for now

Ack.

Thanks,
-Andrea

Quoted 130 lines
Quoted 130 lines

Moreover, apply to find_energy_efficient_cpu() the same SMT-aware preference as the non-EAS wakeup path: when SMT is active and there is a fully-idle core in the relevant domain, prefer max-spare-capacity candidates on fully-idle cores. Otherwise, fall back to the prior behavior, to include also partially-idle SMT siblings.

Cc Vincent Guittot <[email protected]>
Cc Dietmar Eggemann <[email protected]>
Cc Christian Loehle <[email protected]>
Cc Koba Ko <[email protected]>
---
 kernel/sched/fair.c     | 50 +++++++++++++++++++++++++++++++++++++++--
 kernel/sched/topology.c |  9 --------
 2 files changed, 48 insertions(+), 11 deletions(-)
kernel/sched/fair.c +48 -2
index f8deaaa5bfc85..593a89f688679 100644
@@ -8658,13 +8658,15 @@ static int find_energy_efficient_cpu(struct task_struct *p, int prev_cpu)
        eenv_task_busy_time(&eenv, p, prev_cpu);
 
        for (; pd; pd = pd->next) {
-               unsigned long util_min = p_util_min, util_max = p_util_max;
                unsigned long cpu_cap, cpu_actual_cap, util;
                long prev_spare_cap = -1, max_spare_cap = -1;
+               long max_spare_cap_fallback = -1;
                unsigned long rq_util_min, rq_util_max;
                unsigned long cur_delta, base_energy;
-               int max_spare_cap_cpu = -1;
+               int max_spare_cap_cpu = -1, max_spare_cap_cpu_fallback = -1;
                int fits, max_fits = -1;
+               int max_fits_fallback = -1;
+               bool prefer_idle_cores;
 
                if (!cpumask_and(cpus, perf_domain_span(pd), cpu_online_mask))
                        continue;
@@ -8676,6 +8678,8 @@ static int find_energy_efficient_cpu(struct task_struct *p, int prev_cpu)
                eenv.cpu_cap = cpu_actual_cap;
                eenv.pd_cap = 0;
 
+               prefer_idle_cores = sched_smt_active() && test_idle_cores(prev_cpu);
+
                for_each_cpu(cpu, cpus) {
                        struct rq *rq = cpu_rq(cpu);
 
@@ -8687,6 +8691,11 @@ static int find_energy_efficient_cpu(struct task_struct *p, int prev_cpu)
                        if (!cpumask_test_cpu(cpu, p->cpus_ptr))
                                continue;
 
+                       if (prefer_idle_cores && cpu != prev_cpu && !is_core_idle(cpu))
+                               goto fallback;
+
+                       unsigned long util_min = p_util_min, util_max = p_util_max;
+
                        util = cpu_util(cpu, p, cpu, 0);
                        cpu_cap = capacity_of(cpu);
 
@@ -8733,6 +8742,43 @@ static int find_energy_efficient_cpu(struct task_struct *p, int prev_cpu)
                                max_spare_cap_cpu = cpu;
                                max_fits = fits;
                        }
+
+fallback:
+                       if (!prefer_idle_cores || cpu == prev_cpu || is_core_idle(cpu))
+                               continue;
+
+                       util_min = p_util_min;
+                       util_max = p_util_max;
+                       util = cpu_util(cpu, p, cpu, 0);
+                       cpu_cap = capacity_of(cpu);
+
+                       if (uclamp_is_used() && !uclamp_rq_is_idle(rq)) {
+                               rq_util_min = uclamp_rq_get(rq, UCLAMP_MIN);
+                               rq_util_max = uclamp_rq_get(rq, UCLAMP_MAX);
+
+                               util_min = max(rq_util_min, p_util_min);
+                               util_max = max(rq_util_max, p_util_max);
+                       }
+
+                       fits = util_fits_cpu(util, util_min, util_max, cpu);
+                       if (!fits)
+                               continue;
+
+                       lsub_positive(&cpu_cap, util);
+
+                       if ((fits > max_fits_fallback) ||
+                           ((fits == max_fits_fallback) &&
+                            ((long)cpu_cap > max_spare_cap_fallback))) {
+                               max_spare_cap_fallback = cpu_cap;
+                               max_spare_cap_cpu_fallback = cpu;
+                               max_fits_fallback = fits;
+                       }
+               }
+
+               if (max_spare_cap_cpu < 0 && max_spare_cap_cpu_fallback >= 0) {
+                       max_spare_cap = max_spare_cap_fallback;
+                       max_spare_cap_cpu = max_spare_cap_cpu_fallback;
+                       max_fits = max_fits_fallback;
                }
 
                if (max_spare_cap_cpu < 0 && prev_spare_cap < 0)
kernel/sched/topology.c +0 -10
index 061f8c85f5552..cb060fe56aec1 100644
@@ -232,15 +232,6 @@ static bool sched_is_eas_possible(const struct cpumask *cpu_mask)
                return false;
        }
 
-       /* EAS definitely does *not* handle SMT */
-       if (sched_smt_active()) {
-               if (sched_debug()) {
-                       pr_info("rd %*pbl: Checking EAS, SMT is not supported\n",
-                               cpumask_pr_args(cpu_mask));
-               }
-               return false;
-       }
-
        if (!arch_scale_freq_invariant()) {
                if (sched_debug()) {
                        pr_info("rd %*pbl: Checking EAS: frequency-invariant load tracking not yet supported",
--

2.53.0

#16 lore raw

PATCH 4/4 sched/fair: Prefer fully-idle SMT core for NOHZ idle load balancer

In reply to Andrea Righi

When choosing which idle housekeeping CPU runs the idle load balancer, prefer one on a fully idle core if SMT is active, so balance can migrate work onto a CPU that still offers full effective capacity. Fall back to any idle candidate if none qualify.

Cc Vincent Guittot <[email protected]>
Cc Dietmar Eggemann <[email protected]>
Cc Christian Loehle <[email protected]>
Cc Koba Ko <[email protected]>
---
 kernel/sched/fair.c | 19 +++++++++++++++++--
 1 file changed, 17 insertions(+), 2 deletions(-)
kernel/sched/fair.c +17 -2
index 593a89f688679..a1ee21f7b32f6 100644
@@ -12733,11 +12733,15 @@ static inline int on_null_domain(struct rq *rq)
  * - When one of the busy CPUs notices that there may be an idle rebalancing
  *   needed, they will kick the idle load balancer, which then does idle
  *   load balancing for all the idle CPUs.
+ *
+ * - When SMT is active, prefer a CPU on a fully idle core as the ILB
+ *   target, so that when it runs balance it becomes the destination CPU
+ *   and can accept migrated tasks with full effective capacity.
  */
 static inline int find_new_ilb(void)
 {
 	const struct cpumask *hk_mask;
-	int ilb_cpu;
+	int ilb_cpu, fallback = -1;
 
 	hk_mask = housekeeping_cpumask(HK_TYPE_KERNEL_NOISE);
 
@@ -12746,11 +12750,22 @@ static inline int find_new_ilb(void)
 		if (ilb_cpu == smp_processor_id())
 			continue;
 
+#ifdef CONFIG_SCHED_SMT
+		if (!idle_cpu(ilb_cpu))
+			continue;
+
+		if (fallback < 0)
+			fallback = ilb_cpu;
+
+		if (!sched_smt_active() || is_core_idle(ilb_cpu))
+			return ilb_cpu;
+#else
 		if (idle_cpu(ilb_cpu))
 			return ilb_cpu;
+#endif
 	}
 
-	return -1;
+	return fallback;
 }
 
 /*
Signature · 1 line
2.53.0
maintainer
#17 lore raw

Re: [PATCH 4/4] sched/fair: Prefer fully-idle SMT core for NOHZ idle load balancer

Quoted 5 lines · Andrea Righi

When choosing which idle housekeeping CPU runs the idle load balancer, prefer one on a fully idle core if SMT is active, so balance can migrate work onto a CPU that still offers full effective capacity. Fall back to any idle candidate if none qualify.

This one isn't straightforward for me. The ilb cpu will check all other idle CPUs 1st and finish with itself so unless the next CPU in the idle_cpus_mask is a sibling, this should not make a difference

Did you see any perf diff ?

Quoted 33 lines
Cc Vincent Guittot <[email protected]>
Cc Dietmar Eggemann <[email protected]>
Cc Christian Loehle <[email protected]>
Cc Koba Ko <[email protected]>
---
 kernel/sched/fair.c | 19 +++++++++++++++++--
 1 file changed, 17 insertions(+), 2 deletions(-)
kernel/sched/fair.c +5 -1
index 593a89f688679..a1ee21f7b32f6 100644
@@ -12733,11 +12733,15 @@ static inline int on_null_domain(struct rq *rq)
  * - When one of the busy CPUs notices that there may be an idle rebalancing
  *   needed, they will kick the idle load balancer, which then does idle
  *   load balancing for all the idle CPUs.
+ *
+ * - When SMT is active, prefer a CPU on a fully idle core as the ILB
+ *   target, so that when it runs balance it becomes the destination CPU
+ *   and can accept migrated tasks with full effective capacity.
  */
 static inline int find_new_ilb(void)
 {
        const struct cpumask *hk_mask;
-       int ilb_cpu;
+       int ilb_cpu, fallback = -1;
 
        hk_mask = housekeeping_cpumask(HK_TYPE_KERNEL_NOISE);
 

kernel/sched/fair.c static inline int find_new_ilb(void) · line 12753

Quoted 4 lines
kernel/sched/fair.c +1 -0
@@ -12746,11 +12750,22 @@ static inline int find_new_ilb(void)
                if (ilb_cpu == smp_processor_id())
                        continue;
 
+#ifdef CONFIG_SCHED_SMT

you can probably get rid of the CONFIG and put this special case below sched_smt_active()

Quoted 22 lines
kernel/sched/fair.c +11 -2
+               if (!idle_cpu(ilb_cpu))
+                       continue;
+
+               if (fallback < 0)
+                       fallback = ilb_cpu;
+
+               if (!sched_smt_active() || is_core_idle(ilb_cpu))
+                       return ilb_cpu;
+#else
                if (idle_cpu(ilb_cpu))
                        return ilb_cpu;
+#endif
        }
 
-       return -1;
+       return fallback;
 }
 
 /*
--

2.53.0

#18 lore raw

Re: [PATCH 4/4] sched/fair: Prefer fully-idle SMT core for NOHZ idle load balancer

Hi Vincent,

Quoted 7 lines · Vincent Guittot
Quoted 5 lines · Andrea Righi

When choosing which idle housekeeping CPU runs the idle load balancer, prefer one on a fully idle core if SMT is active, so balance can migrate work onto a CPU that still offers full effective capacity. Fall back to any idle candidate if none qualify.

Quoted 5 lines

This one isn't straightforward for me. The ilb cpu will check all other idle CPUs 1st and finish with itself so unless the next CPU in the idle_cpus_mask is a sibling, this should not make a difference

Did you see any perf diff ?

I actually see a benefit, in particular, with the first patch applied I see a ~1.76x speedup, if I add this on top I get ~1.9x speedup vs baseline, which seems pretty consistent across runs (definitely not in error range).

The intention with this change was to minimize SMT noise running the ILB code on a fully-idle core when possible, but I also didn't expect to see such big difference.

I'll investigate more to better understand what's happening.

Quoted 34 lines
Quoted 34 lines
Cc Vincent Guittot <[email protected]>
Cc Dietmar Eggemann <[email protected]>
Cc Christian Loehle <[email protected]>
Cc Koba Ko <[email protected]>
---
 kernel/sched/fair.c | 19 +++++++++++++++++--
 1 file changed, 17 insertions(+), 2 deletions(-)
kernel/sched/fair.c +5 -1
index 593a89f688679..a1ee21f7b32f6 100644
@@ -12733,11 +12733,15 @@ static inline int on_null_domain(struct rq *rq)
  * - When one of the busy CPUs notices that there may be an idle rebalancing
  *   needed, they will kick the idle load balancer, which then does idle
  *   load balancing for all the idle CPUs.
+ *
+ * - When SMT is active, prefer a CPU on a fully idle core as the ILB
+ *   target, so that when it runs balance it becomes the destination CPU
+ *   and can accept migrated tasks with full effective capacity.
  */
 static inline int find_new_ilb(void)
 {
        const struct cpumask *hk_mask;
-       int ilb_cpu;
+       int ilb_cpu, fallback = -1;
 
        hk_mask = housekeeping_cpumask(HK_TYPE_KERNEL_NOISE);
 
Quoted 8 lines

kernel/sched/fair.c static inline int find_new_ilb(void) · line 12753

Quoted 5 lines
kernel/sched/fair.c +1 -0
@@ -12746,11 +12750,22 @@ static inline int find_new_ilb(void)
                if (ilb_cpu == smp_processor_id())
                        continue;
 
+#ifdef CONFIG_SCHED_SMT

you can probably get rid of the CONFIG and put this special case below sched_smt_active()

Ah good point, will change this.

Quoted 16 lines
Quoted 16 lines
kernel/sched/fair.c +10 -0
+               if (!idle_cpu(ilb_cpu))
+                       continue;
+
+               if (fallback < 0)
+                       fallback = ilb_cpu;
+
+               if (!sched_smt_active() || is_core_idle(ilb_cpu))
+                       return ilb_cpu;
+#else
                if (idle_cpu(ilb_cpu))
                        return ilb_cpu;
+#endif
        }
 
Quoted 8 lines
Quoted 8 lines
kernel/sched/fair.c +1 -2
-       return -1;
+       return fallback;
 }
 
 /*
--

2.53.0

Thanks,
-Andrea

#19 lore raw

Re: [PATCH 4/4] sched/fair: Prefer fully-idle SMT core for NOHZ idle load balancer

Hello Andrea,

Quoted 20 lines · Andrea Righi

Hi Vincent,

Quoted 7 lines · Vincent Guittot
Quoted 5 lines · Andrea Righi

When choosing which idle housekeeping CPU runs the idle load balancer, prefer one on a fully idle core if SMT is active, so balance can migrate work onto a CPU that still offers full effective capacity. Fall back to any idle candidate if none qualify.

Quoted 5 lines

This one isn't straightforward for me. The ilb cpu will check all other idle CPUs 1st and finish with itself so unless the next CPU in the idle_cpus_mask is a sibling, this should not make a difference

Did you see any perf diff ?

I actually see a benefit, in particular, with the first patch applied I see a ~1.76x speedup, if I add this on top I get ~1.9x speedup vs baseline, which seems pretty consistent across runs (definitely not in error range).

Quoted 5 lines

The intention with this change was to minimize SMT noise running the ILB code on a fully-idle core when possible, but I also didn't expect to see such big difference.

I'll investigate more to better understand what's happening.

Interesting! Either this "CPU-intensive workload" hates SMT turning busy (but to an extent where performance drops visibly?) or ILB keeps getting interrupted on an SMT sibling that is burdened by interrupts leading to slower balance (or IRQs driving the workload being delayed by rq_lock disabling them)

Would it be possible to share the total SCHED_SOFTIRQ time, load balancing attempts, and utlization with and without the patch? I too will go queue up some runs to see if this makes a difference.

Signature · 2 lines
Thanks and Regards,
Prateek
#20 lore raw

Re: [PATCH 4/4] sched/fair: Prefer fully-idle SMT core for NOHZ idle load balancer

Quoted 35 lines · K Prateek Nayak

Hello Andrea,

Quoted 20 lines · Andrea Righi

Hi Vincent,

Quoted 7 lines · Vincent Guittot
Quoted 5 lines · Andrea Righi

When choosing which idle housekeeping CPU runs the idle load balancer, prefer one on a fully idle core if SMT is active, so balance can migrate work onto a CPU that still offers full effective capacity. Fall back to any idle candidate if none qualify.

Quoted 5 lines

This one isn't straightforward for me. The ilb cpu will check all other idle CPUs 1st and finish with itself so unless the next CPU in the idle_cpus_mask is a sibling, this should not make a difference

Did you see any perf diff ?

I actually see a benefit, in particular, with the first patch applied I see a ~1.76x speedup, if I add this on top I get ~1.9x speedup vs baseline, which seems pretty consistent across runs (definitely not in error range).

Quoted 5 lines

The intention with this change was to minimize SMT noise running the ILB code on a fully-idle core when possible, but I also didn't expect to see such big difference.

I'll investigate more to better understand what's happening.

Interesting! Either this "CPU-intensive workload" hates SMT turning busy (but to an extent where performance drops visibly?) or ILB keeps getting interrupted on an SMT sibling that is burdened by interrupts leading to slower balance (or IRQs driving the workload being delayed by rq_lock disabling them)

Quoted 3 lines

Would it be possible to share the total SCHED_SOFTIRQ time, load balancing attempts, and utlization with and without the patch? I too will go queue up some runs to see if this makes a difference.

Quick update: I also tried this on a Vera machine with a firmware that exposes the same capacity for all the CPUs (so with SD_ASYM_CPUCAPACITY disabled and SMT still on of course) and I see similar performance benefits.

Looking at SCHED_SOFTIRQ and load balancing attempts I don't see big differences, all within error range (results produced using a vibe-coded python script):

- baseline (stats/sec):

SCHED softirq count : 2,625
LB attempts (total) : 69,832

Per-domain breakdown:

    domain0 (SMT):
      lb_count    (total)  :       68,482  [balanced=68,472  failed=9]
        CPU_IDLE         : lb=1,408  imb(load=0 util=0 task=0 misfit=0)  gained=0
        CPU_NEWLY_IDLE   : lb=67,041  imb(load=0 util=0 task=7 misfit=0)  gained=0
        CPU_NOT_IDLE     : lb=33  imb(load=0 util=0 task=2 misfit=0)  gained=0
    domain1 (MC):
      lb_count    (total)  :          902  [balanced=900  failed=2]
        CPU_NEWLY_IDLE   : lb=869  imb(load=0 util=0 task=0 misfit=0)  gained=0
        CPU_NOT_IDLE     : lb=33  imb(load=0 util=0 task=2 misfit=0)  gained=0
    domain2 (NUMA):
      lb_count    (total)  :          448  [balanced=441  failed=7]
        CPU_NEWLY_IDLE   : lb=415  imb(load=0 util=0 task=44 misfit=0)  gained=0
        CPU_NOT_IDLE     : lb=33  imb(load=0 util=0 task=268 misfit=0)  gained=0

- with ilb-smt (stats/sec):

SCHED softirq count : 2,671
LB attempts (total) : 68,572

Per-domain breakdown:

    domain0 (SMT):
      lb_count    (total)  :       67,239  [balanced=67,197  failed=41]
        CPU_IDLE         : lb=1,419  imb(load=0 util=0 task=0 misfit=0)  gained=0
        CPU_NEWLY_IDLE   : lb=65,783  imb(load=0 util=0 task=42 misfit=0)  gained=1
        CPU_NOT_IDLE     : lb=37  imb(load=0 util=0 task=0 misfit=0)  gained=0
    domain1 (MC):
      lb_count    (total)  :          833  [balanced=833  failed=0]
        CPU_NEWLY_IDLE   : lb=796  imb(load=0 util=0 task=0 misfit=0)  gained=0
        CPU_NOT_IDLE     : lb=37  imb(load=0 util=0 task=0 misfit=0)  gained=0
    domain2 (NUMA):
      lb_count    (total)  :          500  [balanced=488  failed=12]
        CPU_NEWLY_IDLE   : lb=463  imb(load=0 util=0 task=44 misfit=0)  gained=0
        CPU_NOT_IDLE     : lb=37  imb(load=0 util=0 task=627 misfit=0)  gained=0

I'll add more direct instrumentation to check what ILB is doing differently...

And I'll also repeat the test and collect the same metrics on the Vera machine with the firmware that exposes different CPU capacities as soon as I get access again.

Thanks,
-Andrea

#21 lore raw

Re: [PATCH 4/4] sched/fair: Prefer fully-idle SMT core for NOHZ idle load balancer

Quoted 89 lines · Andrea Righi
Quoted 35 lines · K Prateek Nayak

Hello Andrea,

Quoted 20 lines · Andrea Righi

Hi Vincent,

Quoted 7 lines · Vincent Guittot
Quoted 5 lines · Andrea Righi

When choosing which idle housekeeping CPU runs the idle load balancer, prefer one on a fully idle core if SMT is active, so balance can migrate work onto a CPU that still offers full effective capacity. Fall back to any idle candidate if none qualify.

Quoted 5 lines

This one isn't straightforward for me. The ilb cpu will check all other idle CPUs 1st and finish with itself so unless the next CPU in the idle_cpus_mask is a sibling, this should not make a difference

Did you see any perf diff ?

I actually see a benefit, in particular, with the first patch applied I see a ~1.76x speedup, if I add this on top I get ~1.9x speedup vs baseline, which seems pretty consistent across runs (definitely not in error range).

Quoted 5 lines

The intention with this change was to minimize SMT noise running the ILB code on a fully-idle core when possible, but I also didn't expect to see such big difference.

I'll investigate more to better understand what's happening.

Interesting! Either this "CPU-intensive workload" hates SMT turning busy (but to an extent where performance drops visibly?) or ILB keeps getting interrupted on an SMT sibling that is burdened by interrupts leading to slower balance (or IRQs driving the workload being delayed by rq_lock disabling them)

Quoted 3 lines

Would it be possible to share the total SCHED_SOFTIRQ time, load balancing attempts, and utlization with and without the patch? I too will go queue up some runs to see if this makes a difference.

Quick update: I also tried this on a Vera machine with a firmware that exposes the same capacity for all the CPUs (so with SD_ASYM_CPUCAPACITY disabled and SMT still on of course) and I see similar performance benefits.

Looking at SCHED_SOFTIRQ and load balancing attempts I don't see big differences, all within error range (results produced using a vibe-coded python script):

- baseline (stats/sec):

SCHED softirq count : 2,625
LB attempts (total) : 69,832

Per-domain breakdown:

    domain0 (SMT):
      lb_count    (total)  :       68,482  [balanced=68,472  failed=9]
        CPU_IDLE         : lb=1,408  imb(load=0 util=0 task=0 misfit=0)  gained=0
        CPU_NEWLY_IDLE   : lb=67,041  imb(load=0 util=0 task=7 misfit=0)  gained=0
        CPU_NOT_IDLE     : lb=33  imb(load=0 util=0 task=2 misfit=0)  gained=0
    domain1 (MC):
      lb_count    (total)  :          902  [balanced=900  failed=2]
        CPU_NEWLY_IDLE   : lb=869  imb(load=0 util=0 task=0 misfit=0)  gained=0
        CPU_NOT_IDLE     : lb=33  imb(load=0 util=0 task=2 misfit=0)  gained=0
    domain2 (NUMA):
      lb_count    (total)  :          448  [balanced=441  failed=7]
        CPU_NEWLY_IDLE   : lb=415  imb(load=0 util=0 task=44 misfit=0)  gained=0
        CPU_NOT_IDLE     : lb=33  imb(load=0 util=0 task=268 misfit=0)  gained=0

- with ilb-smt (stats/sec):

SCHED softirq count : 2,671
LB attempts (total) : 68,572

Per-domain breakdown:

    domain0 (SMT):
      lb_count    (total)  :       67,239  [balanced=67,197  failed=41]
        CPU_IDLE         : lb=1,419  imb(load=0 util=0 task=0 misfit=0)  gained=0
        CPU_NEWLY_IDLE   : lb=65,783  imb(load=0 util=0 task=42 misfit=0)  gained=1
        CPU_NOT_IDLE     : lb=37  imb(load=0 util=0 task=0 misfit=0)  gained=0
    domain1 (MC):
      lb_count    (total)  :          833  [balanced=833  failed=0]
        CPU_NEWLY_IDLE   : lb=796  imb(load=0 util=0 task=0 misfit=0)  gained=0
        CPU_NOT_IDLE     : lb=37  imb(load=0 util=0 task=0 misfit=0)  gained=0
    domain2 (NUMA):
      lb_count    (total)  :          500  [balanced=488  failed=12]
        CPU_NEWLY_IDLE   : lb=463  imb(load=0 util=0 task=44 misfit=0)  gained=0
        CPU_NOT_IDLE     : lb=37  imb(load=0 util=0 task=627 misfit=0)  gained=0
Quoted 2 lines

I'll add more direct instrumentation to check what ILB is doing differently...

More data.

== SMT contention ==

tracepoint:sched:sched_switch

{
    if (args->next_pid != 0) {
        @busy[cpu] = 1;
    } else {
        delete(@busy[cpu]);
    }
}

tracepoint:sched:sched_switch
/ args->prev_pid == 0 && args->next_pid != 0 /

{
    $sib = (cpu + 176) % 352;

    if (@busy[$sib]) {
        @smt_contention++;
    } else {
        @smt_no_contention++;
    }
}

END

{
    printf("smt_contention %lld\n", (int64)@smt_contention);
    printf("smt_no_contention %lld\n", (int64)@smt_no_contention);
}

- baseline:

@smt_contention: 1103
@smt_no_contention: 3815

- ilb-smt:

@smt_contention: 937
@smt_no_contention: 4459

== ILB duration ==

- baseline:

@ilb_duration_us:

[0]                  147 |                                                    |
[1]                  354 |@                                                   |
[2, 4)               739 |@@@                                                 |
[4, 8)              3040 |@@@@@@@@@@@@@@@@                                    |
[8, 16)             9825 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[16, 32)            8142 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@         |
[32, 64)            1267 |@@@@@@                                              |
[64, 128)           1607 |@@@@@@@@                                            |
[128, 256)          2222 |@@@@@@@@@@@                                         |
[256, 512)          2326 |@@@@@@@@@@@@                                        |
[512, 1K)            141 |                                                    |
[1K, 2K)              37 |                                                    |
[2K, 4K)               7 |

- ilb-smt:

@ilb_duration_us:

[0]                   79 |                                                    |
[1]                  137 |                                                    |
[2, 4)              1440 |@@@@@@@@@@                                          |
[4, 8)              2897 |@@@@@@@@@@@@@@@@@@@@                                |
[8, 16)             7433 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[16, 32)            4993 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@                  |
[32, 64)            2390 |@@@@@@@@@@@@@@@@                                    |
[64, 128)           2254 |@@@@@@@@@@@@@@@                                     |
[128, 256)          2731 |@@@@@@@@@@@@@@@@@@@                                 |
[256, 512)          1083 |@@@@@@@                                             |
[512, 1K)            265 |@                                                   |
[1K, 2K)              29 |                                                    |
[2K, 4K)               5 |                                                    |

== rq_lock hold ==

- baseline:

@lb_rqlock_hold_us:

[0]               664396 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[1]                77446 |@@@@@@                                              |
[2, 4)             25044 |@                                                   |
[4, 8)             19847 |@                                                   |
[8, 16)             2434 |                                                    |
[16, 32)             605 |                                                    |
[32, 64)             308 |                                                    |
[64, 128)             38 |                                                    |
[128, 256)             2 |                                                    |

- ilb-smt:

@lb_rqlock_hold_us:

[0]               229152 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[1]               135060 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@                      |
[2, 4)             26989 |@@@@@@                                              |
[4, 8)             48034 |@@@@@@@@@@                                          |
[8, 16)             1919 |                                                    |
[16, 32)            2236 |                                                    |
[32, 64)             595 |                                                    |
[64, 128)            135 |                                                    |
[128, 256)            27 |                                                    |

For what I see ILB runs are more expensive, but I still don't see why I'm getting the speedup with this ilb-smt patch. I'll keep investigating...

-Andrea

#22 lore raw

Re: [PATCH 4/4] sched/fair: Prefer fully-idle SMT core for NOHZ idle load balancer

In reply to K Prateek Nayak
Quoted 27 lines · K Prateek Nayak

Hello Andrea,

Quoted 24 lines · Andrea Righi

Hi Vincent,

Quoted 7 lines · Vincent Guittot
Quoted 5 lines · Andrea Righi

When choosing which idle housekeeping CPU runs the idle load balancer, prefer one on a fully idle core if SMT is active, so balance can migrate work onto a CPU that still offers full effective capacity. Fall back to any idle candidate if none qualify.

Quoted 5 lines

This one isn't straightforward for me. The ilb cpu will check all other idle CPUs 1st and finish with itself so unless the next CPU in the idle_cpus_mask is a sibling, this should not make a difference

Did you see any perf diff ?

I actually see a benefit, in particular, with the first patch applied I see a ~1.76x speedup, if I add this on top I get ~1.9x speedup vs baseline, which seems pretty consistent across runs (definitely not in error range).

The intention with this change was to minimize SMT noise running the ILB code on a fully-idle core when possible, but I also didn't expect to see such big difference.

Quoted 7 lines
Quoted 1 line

I'll investigate more to better understand what's happening.

Interesting! Either this "CPU-intensive workload" hates SMT turning busy (but to an extent where performance drops visibly?) or ILB keeps getting interrupted on an SMT sibling that is burdened by interrupts leading to slower balance (or IRQs driving the workload being delayed by rq_lock disabling them)

Alright, I dug a bit deeper into what's going on.

In this case, the workload showing the large benefit (the NVBLAS benchmark) is running exactly one task per SMT core, all pinned to NUMA node 0. The system has two nodes, so node 1 remains mostly idle.

With the SMT-aware select_idle_capacity(), tasks get distributed across SMT cores in a way that avoids placing them on busy siblings, which is nice and it's the part that gives most of the speedup.

However, without this ILB patch, find_new_ilb() always picks a CPU with a busy sibling on node 0, because for_each_cpu_and() always starts from the lower CPU IDs. As a result, the ILB always ends up running on CPUs with a CPU-intensive worker running on its sibling, disrupting each other's performance.

As an experiment, I tried something silly like the following, biasing the ILB selection toward node 1 (node0 = 0-87,176-263, node1 = 88-177,264-351):

	struct cpumask tmp;

	cpumask_and(&tmp, nohz.idle_cpus_mask, hk_mask);
	for_each_cpu_wrap(ilb_cpu, &tmp, nr_cpu_ids / 4) {
		if (ilb_cpu == smp_processor_id())
			continue;

		if (idle_cpu(ilb_cpu))
			return ilb_cpu;
	}

And I get pretty much the same speedup (slighly better actually, because I always get an idle CPU in one step, since node 1 is always idle with this particular benchmark).

So, in this particular scenario this patch makes sense, because we avoid the "SMT contention" at very low cost. In general, I think the benefit can be quite situational. I could still make sense to have it, the extra overhead is limited to an additional is_core_idle() check over idle & HK candidates (worst case), which could be worthwhile if it reduces interference from busy SMT siblings.

What do you think?

Thanks,
-Andrea

#23 lore raw

Re: [PATCH 4/4] sched/fair: Prefer fully-idle SMT core for NOHZ idle load balancer

In reply to Andrea Righi
Quoted 38 lines · Andrea Righi

When choosing which idle housekeeping CPU runs the idle load balancer, prefer one on a fully idle core if SMT is active, so balance can migrate work onto a CPU that still offers full effective capacity. Fall back to any idle candidate if none qualify.

Cc Vincent Guittot <[email protected]>
Cc Dietmar Eggemann <[email protected]>
Cc Christian Loehle <[email protected]>
Cc Koba Ko <[email protected]>
---
  kernel/sched/fair.c | 19 +++++++++++++++++--
  1 file changed, 17 insertions(+), 2 deletions(-)
kernel/sched/fair.c +5 -1
index 593a89f688679..a1ee21f7b32f6 100644
@@ -12733,11 +12733,15 @@ static inline int on_null_domain(struct rq *rq)
   * - When one of the busy CPUs notices that there may be an idle rebalancing
   *   needed, they will kick the idle load balancer, which then does idle
   *   load balancing for all the idle CPUs.
+ *
+ * - When SMT is active, prefer a CPU on a fully idle core as the ILB
+ *   target, so that when it runs balance it becomes the destination CPU
+ *   and can accept migrated tasks with full effective capacity.
   */
  static inline int find_new_ilb(void)
  {
  	const struct cpumask *hk_mask;
-	int ilb_cpu;
+	int ilb_cpu, fallback = -1;
 
  	hk_mask = housekeeping_cpumask(HK_TYPE_KERNEL_NOISE);
 
@@ -12746,11 +12750,22 @@ static inline int find_new_ilb(void)
  		if (ilb_cpu == smp_processor_id())

kernel/sched/fair.c static inline int find_new_ilb(void) · line 12760

Quoted 10 lines
kernel/sched/fair.c +8 -0
  			continue;
 
+#ifdef CONFIG_SCHED_SMT
+		if (!idle_cpu(ilb_cpu))
+			continue;
+
+		if (fallback < 0)
+			fallback = ilb_cpu;
+
+		if (!sched_smt_active() || is_core_idle(ilb_cpu))

is_core_idle does loop for all sublings and nohz.idle_cpus_mask will have all siblings likely.

So that might turn out be a bit expensive on large SMT system such as SMT=4 Also, this is with interrupt disabled.

Will try to run this on powerpc system and see if simple benchmarks show anything.

Quoted 12 lines
kernel/sched/fair.c +4 -1
+			return ilb_cpu;
+#else
  		if (idle_cpu(ilb_cpu))
  			return ilb_cpu;
+#endif
  	}
 
-	return -1;
+	return fallback;
  }
 
  /*
#24 lore raw
In reply to Andrea Righi
Quoted 30 lines · Andrea Righi

This series attempts to improve SD_ASYM_CPUCAPACITY scheduling by introducing SMT awareness.

= Problem =

Nominal per-logical-CPU capacity can overstate usable compute when an SMT sibling is busy, because the physical core doesn't deliver its full nominal capacity. So, several SD_ASYM_CPUCAPACITY paths may pick high capacity CPUs that are not actually good destinations.

= Proposed Solution =

This patch set aligns those paths with a simple rule already used
elsewhere: when SMT is active, prefer fully idle cores and avoid treating partially idle SMT siblings as full-capacity targets where that would mislead load balance.

Patch set summary:

- [PATCH 1/4] sched/fair: Prefer fully-idle SMT cores in asym-capacity idle selection

Prefer fully-idle SMT cores in asym-capacity idle selection. In the wakeup fast path, extend select_idle_capacity() / asym_fits_cpu() so idle selection can prefer CPUs on fully idle cores, with a safe fallback.

- [PATCH 2/4] sched/fair: Reject misfit pulls onto busy SMT siblings on asym-capacity

Reject misfit pulls onto busy SMT siblings on SD_ASYM_CPUCAPACITY. Provided for consistency with PATCH 1/4.

Quoted 6 lines

- [PATCH 3/4] sched/fair: Enable EAS with SMT on SD_ASYM_CPUCAPACITY systems

Enable EAS with SD_ASYM_CPUCAPACITY and SMT. Also provided for consistency with PATCH 1/4. I've also tested with/without /proc/sys/kernel/sched_energy_aware enabled (same platform) and haven't noticed any regression.

There's a lot more to unpack, but just to confirm, Vera doesn't have an EM, right? There's no EAS with it?
(To be more precise, CPPC should bail out of building an artifical EM if there's no or only one efficiency class:
drivers/cpufreq/cppc_cpufreq.c:

if (bitmap_weight(used_classes, 256) <= 1) {
		pr_debug("Efficiency classes are all equal (=%d). "
			"No EM registered", class);
		return;
	}

This is the case, right?

Quoted 1 line

[snip]

#25 lore raw
Quoted 45 lines · Christian Loehle
Quoted 30 lines · Andrea Righi

This series attempts to improve SD_ASYM_CPUCAPACITY scheduling by introducing SMT awareness.

= Problem =

Nominal per-logical-CPU capacity can overstate usable compute when an SMT sibling is busy, because the physical core doesn't deliver its full nominal capacity. So, several SD_ASYM_CPUCAPACITY paths may pick high capacity CPUs that are not actually good destinations.

= Proposed Solution =

This patch set aligns those paths with a simple rule already used
elsewhere: when SMT is active, prefer fully idle cores and avoid treating partially idle SMT siblings as full-capacity targets where that would mislead load balance.

Patch set summary:

- [PATCH 1/4] sched/fair: Prefer fully-idle SMT cores in asym-capacity idle selection

Prefer fully-idle SMT cores in asym-capacity idle selection. In the wakeup fast path, extend select_idle_capacity() / asym_fits_cpu() so idle selection can prefer CPUs on fully idle cores, with a safe fallback.

- [PATCH 2/4] sched/fair: Reject misfit pulls onto busy SMT siblings on asym-capacity

Reject misfit pulls onto busy SMT siblings on SD_ASYM_CPUCAPACITY. Provided for consistency with PATCH 1/4.

Quoted 6 lines

- [PATCH 3/4] sched/fair: Enable EAS with SMT on SD_ASYM_CPUCAPACITY systems

Enable EAS with SD_ASYM_CPUCAPACITY and SMT. Also provided for consistency with PATCH 1/4. I've also tested with/without /proc/sys/kernel/sched_energy_aware enabled (same platform) and haven't noticed any regression.

There's a lot more to unpack, but just to confirm, Vera doesn't have an EM, right? There's no EAS with it?
(To be more precise, CPPC should bail out of building an artifical EM if there's no or only one efficiency class:
drivers/cpufreq/cppc_cpufreq.c:

Quoted 7 lines
if (bitmap_weight(used_classes, 256) <= 1) {
		pr_debug("Efficiency classes are all equal (=%d). "
			"No EM registered", class);
		return;
	}

This is the case, right?

Yes, that's correct, so my testing on Vera with EAS isn't that meaningful.

Thanks,
-Andrea

#26 lore raw
In reply to Andrea Righi

Hi Andrea.

Quoted 10 lines · Andrea Righi

This series attempts to improve SD_ASYM_CPUCAPACITY scheduling by introducing SMT awareness.

= Problem =

Nominal per-logical-CPU capacity can overstate usable compute when an SMT sibling is busy, because the physical core doesn't deliver its full nominal capacity. So, several SD_ASYM_CPUCAPACITY paths may pick high capacity CPUs that are not actually good destinations.

How does energy model define the opp for SMT?

SMT systems have multiple of different functional blocks, a few ALU(arithmetic), LSU(load store unit) etc. If same/similar workload runs on sibling, it would affect the performance, but sibling is using different functional blocks, then it would not.

So underlying actual CPU Capacity of each thread depends on what each sibling is running. I don't understand how does the firmware/energy models define this.

Quoted 33 lines

= Proposed Solution =

This patch set aligns those paths with a simple rule already used
elsewhere: when SMT is active, prefer fully idle cores and avoid treating partially idle SMT siblings as full-capacity targets where that would mislead load balance.

Patch set summary:

- [PATCH 1/4] sched/fair: Prefer fully-idle SMT cores in asym-capacity idle selection

Prefer fully-idle SMT cores in asym-capacity idle selection. In the wakeup fast path, extend select_idle_capacity() / asym_fits_cpu() so idle selection can prefer CPUs on fully idle cores, with a safe fallback.

- [PATCH 2/4] sched/fair: Reject misfit pulls onto busy SMT siblings on asym-capacity

Reject misfit pulls onto busy SMT siblings on SD_ASYM_CPUCAPACITY. Provided for consistency with PATCH 1/4.

- [PATCH 3/4] sched/fair: Enable EAS with SMT on SD_ASYM_CPUCAPACITY systems

Enable EAS with SD_ASYM_CPUCAPACITY and SMT. Also provided for consistency with PATCH 1/4. I've also tested with/without /proc/sys/kernel/sched_energy_aware enabled (same platform) and haven't noticed any regression.

- [PATCH 4/4] sched/fair: Prefer fully-idle SMT core for NOHZ idle load balancer

When choosing the housekeeping CPU that runs the idle load balancer, prefer an idle CPU on a fully idle core so migrated work lands where effective capacity is available.

Quoted 9 lines

The change is still consistent with the same "avoid CPUs with busy sibling" logic and it shows some benefits on Vera, but could have negative impact on other systems, I'm including it for completeness (feedback is appreciated).

This patch set has been tested on the new NVIDIA Vera Rubin platform, where SMT is enabled and the firmware exposes small frequency variations (+/-~5%) as differences in CPU capacity, resulting in SD_ASYM_CPUCAPACITY being set.

I assume the CPU_CAPACITY values fixed?
first sibling has max, while other has less?

Quoted 4 lines

Without these patches, performance can drop up to ~2x with CPU-intensive workloads, because the SD_ASYM_CPUCAPACITY idle selection policy does not account for busy SMT siblings.

How is the performance measured here? Which benchmark?
By any chance you are running number_running_task <= (nr_cpus / smt_threads_per_core), so it is all fitting nicely?

If you increase those numbers, how does the performance numbers compare?

Also, whats the system is like? SMT level?

Quoted 23 lines

Alternative approaches have been evaluated, such as equalizing CPU capacities, either by exposing uniform values via firmware (ACPI/CPPC) or normalizing them in the kernel by grouping CPUs within a small capacity window (+-5%) [1][2], or enabling asympacking [3].

However, adding SMT awareness to SD_ASYM_CPUCAPACITY has shown better results so far. Improving this policy also seems worthwhile in general, as other platforms in the future may enable SMT with asymmetric CPU topologies.

[1] https://lore.kernel.org/lkml/[email protected]
[2] https://lore.kernel.org/lkml/[email protected]
[3] https://lore.kernel.org/all/[email protected]/

Andrea Righi (4):
sched/fair: Prefer fully-idle SMT cores in asym-capacity idle selection sched/fair: Reject misfit pulls onto busy SMT siblings on asym-capacity sched/fair: Enable EAS with SMT on SD_ASYM_CPUCAPACITY systems sched/fair: Prefer fully-idle SMT core for NOHZ idle load balancer

  kernel/sched/fair.c     | 163 +++++++++++++++++++++++++++++++++++++++++++-----
  kernel/sched/topology.c |   9 ---
  2 files changed, 147 insertions(+), 25 deletions(-)
#27 lore raw
Quoted 8 lines · Shrikanth Hegde

Hi Andrea.

Quoted 5 lines · Andrea Righi

This series attempts to improve SD_ASYM_CPUCAPACITY scheduling by introducing SMT awareness.

= Problem =

Quoted 7 lines
Quoted 5 lines

Nominal per-logical-CPU capacity can overstate usable compute when an SMT sibling is busy, because the physical core doesn't deliver its full nominal capacity. So, several SD_ASYM_CPUCAPACITY paths may pick high capacity CPUs that are not actually good destinations.

How does energy model define the opp for SMT?

For now, as suggested by Vincent, we should probably ignore EAS / energy model and keep it as it is (not compatible with SMT). I'll drop PATCH 3/4 and focus only at SD_ASYM_CPUCAPACITY + SMT.

Quoted 8 lines

SMT systems have multiple of different functional blocks, a few ALU(arithmetic), LSU(load store unit) etc. If same/similar workload runs on sibling, it would affect the performance, but sibling is using different functional blocks, then it would not.

So underlying actual CPU Capacity of each thread depends on what each sibling is running. I don't understand how does the firmware/energy models define this.

They don't and they probably shouldn't. I don't think it's possible to model CPU capacity with a static nominal value when SMT is enabled, since the effective capacity changes if the corresponding sibling is busy or not.

It should be up to the scheduler to figure out a reasonable way to estimate the actual capacity, considering the status of the other sibling (e.g., prioritizing the fully-idle SMT cores over the partially-idle SMT cores, like we do in other parts of the scheduler code).

Quoted 39 lines
Quoted 39 lines

= Proposed Solution =

This patch set aligns those paths with a simple rule already used
elsewhere: when SMT is active, prefer fully idle cores and avoid treating partially idle SMT siblings as full-capacity targets where that would mislead load balance.

Patch set summary:

- [PATCH 1/4] sched/fair: Prefer fully-idle SMT cores in asym-capacity idle selection

Prefer fully-idle SMT cores in asym-capacity idle selection. In the wakeup fast path, extend select_idle_capacity() / asym_fits_cpu() so idle selection can prefer CPUs on fully idle cores, with a safe fallback.

- [PATCH 2/4] sched/fair: Reject misfit pulls onto busy SMT siblings on asym-capacity

Reject misfit pulls onto busy SMT siblings on SD_ASYM_CPUCAPACITY. Provided for consistency with PATCH 1/4.

- [PATCH 3/4] sched/fair: Enable EAS with SMT on SD_ASYM_CPUCAPACITY systems

Enable EAS with SD_ASYM_CPUCAPACITY and SMT. Also provided for consistency with PATCH 1/4. I've also tested with/without /proc/sys/kernel/sched_energy_aware enabled (same platform) and haven't noticed any regression.

- [PATCH 4/4] sched/fair: Prefer fully-idle SMT core for NOHZ idle load balancer

When choosing the housekeeping CPU that runs the idle load balancer, prefer an idle CPU on a fully idle core so migrated work lands where effective capacity is available.

The change is still consistent with the same "avoid CPUs with busy sibling" logic and it shows some benefits on Vera, but could have negative impact on other systems, I'm including it for completeness (feedback is appreciated).

Quoted 7 lines
Quoted 4 lines

This patch set has been tested on the new NVIDIA Vera Rubin platform, where SMT is enabled and the firmware exposes small frequency variations (+/-~5%) as differences in CPU capacity, resulting in SD_ASYM_CPUCAPACITY being set.

I assume the CPU_CAPACITY values fixed?
first sibling has max, while other has less?

The firmware is exposing the same capacity for both siblings. SMT cores may have different capacity, but siblings within the same SMT core have the same capacity.

There was an idea to expose a higher capacity for all the 1st sibling and a lower capacity for all the 2nd siblings, but I don't think it's a good idea, since that would just confuse the scheduler (and the 2nd sibling doesn't really have a lower nominal capacity if it's running alone).

Quoted 7 lines
Quoted 5 lines

Without these patches, performance can drop up to ~2x with CPU-intensive workloads, because the SD_ASYM_CPUCAPACITY idle selection policy does not account for busy SMT siblings.

How is the performance measured here? Which benchmark?

I've used an internal NVIDIA suite (based on NVBLAS), I also tried Linpack and got similar results. I'm planning to repeat the tests using public benchmarks and share the results as soon as I can.

Quoted 2 lines

By any chance you are running number_running_task <= (nr_cpus / smt_threads_per_core), so it is all fitting nicely?

That's the case that gives me the optimal results.

Quoted 2 lines

If you increase those numbers, how does the performance numbers compare?

I tried different number of tasks. The more I approach system saturation the smaller the benefits are. When I completely saturate the system I don't see any benefit with this changes, neither regressions, but I guess that's expected.

Quoted 2 lines

Also, whats the system is like? SMT level?

2 siblings for each SMT core.

Thanks,
-Andrea

#28 lore raw
Quoted 13 lines
Quoted 2 lines

How is the performance measured here? Which benchmark?

I've used an internal NVIDIA suite (based on NVBLAS), I also tried Linpack and got similar results. I'm planning to repeat the tests using public benchmarks and share the results as soon as I can.

Quoted 2 lines

By any chance you are running number_running_task <= (nr_cpus / smt_threads_per_core), so it is all fitting nicely?

That's the case that gives me the optimal results.

Quoted 1 line
Quoted 7 lines
Quoted 1 line

If you increase those numbers, how does the performance numbers compare?

I tried different number of tasks. The more I approach system saturation the smaller the benefits are. When I completely saturate the system I don't see any benefit with this changes, neither regressions, but I guess that's expected.

Ok. That's good.

I gave hackbench on powerpc with SMT=4, i didn't observe any regressions or improvements. Only PATCH 4/4 applies in this case as there is no asym_cpu_capacity

#29 lore raw
In reply to Andrea Righi
Quoted 10 lines · Andrea Righi

This series attempts to improve SD_ASYM_CPUCAPACITY scheduling by introducing SMT awareness.

= Problem =

Nominal per-logical-CPU capacity can overstate usable compute when an SMT sibling is busy, because the physical core doesn't deliver its full nominal capacity. So, several SD_ASYM_CPUCAPACITY paths may pick high capacity CPUs that are not actually good destinations.

Quoted 6 lines

= Proposed Solution =

This patch set aligns those paths with a simple rule already used
elsewhere: when SMT is active, prefer fully idle cores and avoid treating partially idle SMT siblings as full-capacity targets where that would mislead load balance.

In kernel/sched/topology.c

	/* Don't attempt to spread across CPUs of different capacities. */
	if ((sd->flags & SD_ASYM_CPUCAPACITY) && sd->child)
		sd->child->flags &= ~SD_PREFER_SIBLING;

Should handle the selection, but I guess this does not work for SMT level sd's?

Quoted 27 lines

Patch set summary:

- [PATCH 1/4] sched/fair: Prefer fully-idle SMT cores in asym-capacity idle selection

Prefer fully-idle SMT cores in asym-capacity idle selection. In the wakeup fast path, extend select_idle_capacity() / asym_fits_cpu() so idle selection can prefer CPUs on fully idle cores, with a safe fallback.

- [PATCH 2/4] sched/fair: Reject misfit pulls onto busy SMT siblings on asym-capacity

Reject misfit pulls onto busy SMT siblings on SD_ASYM_CPUCAPACITY. Provided for consistency with PATCH 1/4.

- [PATCH 3/4] sched/fair: Enable EAS with SMT on SD_ASYM_CPUCAPACITY systems

Enable EAS with SD_ASYM_CPUCAPACITY and SMT. Also provided for consistency with PATCH 1/4. I've also tested with/without /proc/sys/kernel/sched_energy_aware enabled (same platform) and haven't noticed any regression.

- [PATCH 4/4] sched/fair: Prefer fully-idle SMT core for NOHZ idle load balancer

When choosing the housekeeping CPU that runs the idle load balancer, prefer an idle CPU on a fully idle core so migrated work lands where effective capacity is available.

Quoted 9 lines

The change is still consistent with the same "avoid CPUs with busy sibling" logic and it shows some benefits on Vera, but could have negative impact on other systems, I'm including it for completeness (feedback is appreciated).

This patch set has been tested on the new NVIDIA Vera Rubin platform, where SMT is enabled and the firmware exposes small frequency variations (+/-~5%) as differences in CPU capacity, resulting in SD_ASYM_CPUCAPACITY being set.

Are you referring to nominal_freq?

Quoted 24 lines

Without these patches, performance can drop up to ~2x with CPU-intensive workloads, because the SD_ASYM_CPUCAPACITY idle selection policy does not account for busy SMT siblings.

Alternative approaches have been evaluated, such as equalizing CPU capacities, either by exposing uniform values via firmware (ACPI/CPPC) or normalizing them in the kernel by grouping CPUs within a small capacity window (+-5%) [1][2], or enabling asympacking [3].

However, adding SMT awareness to SD_ASYM_CPUCAPACITY has shown better results so far. Improving this policy also seems worthwhile in general, as other platforms in the future may enable SMT with asymmetric CPU topologies.

[1] https://lore.kernel.org/lkml/[email protected]
[2] https://lore.kernel.org/lkml/[email protected]
[3] https://lore.kernel.org/all/[email protected]/

Andrea Righi (4):
sched/fair: Prefer fully-idle SMT cores in asym-capacity idle selection sched/fair: Reject misfit pulls onto busy SMT siblings on asym-capacity sched/fair: Enable EAS with SMT on SD_ASYM_CPUCAPACITY systems sched/fair: Prefer fully-idle SMT core for NOHZ idle load balancer

Quoted 3 lines
 kernel/sched/fair.c     | 163 +++++++++++++++++++++++++++++++++++++++++++-----
 kernel/sched/topology.c |   9 ---
 2 files changed, 147 insertions(+), 25 deletions(-)

Thanks,
Balbir

#30 lore raw

Hi Balbir,

Quoted 18 lines · Balbir Singh
Quoted 10 lines · Andrea Righi

This series attempts to improve SD_ASYM_CPUCAPACITY scheduling by introducing SMT awareness.

= Problem =

Nominal per-logical-CPU capacity can overstate usable compute when an SMT sibling is busy, because the physical core doesn't deliver its full nominal capacity. So, several SD_ASYM_CPUCAPACITY paths may pick high capacity CPUs that are not actually good destinations.

Quoted 6 lines

= Proposed Solution =

This patch set aligns those paths with a simple rule already used
elsewhere: when SMT is active, prefer fully idle cores and avoid treating partially idle SMT siblings as full-capacity targets where that would mislead load balance.

Quoted 7 lines

In kernel/sched/topology.c

	/* Don't attempt to spread across CPUs of different capacities. */
	if ((sd->flags & SD_ASYM_CPUCAPACITY) && sd->child)
		sd->child->flags &= ~SD_PREFER_SIBLING;

Should handle the selection, but I guess this does not work for SMT level sd's?

IIUC, SD_PREFER_SIBLING steers load balance toward sibling_imbalance() (spread runnables across child/sibling domains), it doesn't encode the fully-idle core first logic. In practice it doesn't give us SMT-aware destination choice when a sibling is busy and this series is trying to cover that gap in the palcement path.

BTW, on Vera the hierarchy is SMT -> MC -> NUMA:

root@localhost:~# grep . /sys/kernel/debug/sched/domains/cpu0/domain*/flags
/sys/kernel/debug/sched/domains/cpu0/domain0/flags:SD_BALANCE_NEWIDLE SD_BALANCE_EXEC SD_BALANCE_FORK SD_WAKE_AFFINE SD_SHARE_CPUCAPACITY SD_SHARE_LLC SD_PREFER_SIBLING
/sys/kernel/debug/sched/domains/cpu0/domain1/flags:SD_BALANCE_NEWIDLE SD_BALANCE_EXEC SD_BALANCE_FORK SD_WAKE_AFFINE SD_ASYM_CPUCAPACITY SD_SHARE_LLC
/sys/kernel/debug/sched/domains/cpu0/domain2/flags:SD_BALANCE_NEWIDLE SD_ASYM_CPUCAPACITY SD_ASYM_CPUCAPACITY_FULL SD_SERIALIZE SD_NUMA

And domain1/groups_flags (child / SMT flags on the sched groups used at the MC level) still has SD_PREFER_SIBLING together with SD_SHARE_CPUCAPACITY.

root@localhost:~# cat /sys/kernel/debug/sched/domains/cpu0/domain1/groups_flags
SD_BALANCE_NEWIDLE SD_BALANCE_EXEC SD_BALANCE_FORK SD_WAKE_AFFINE SD_SHARE_CPUCAPACITY SD_SHARE_LLC SD_PREFER_SIBLING

So, prefer-sibling is still in play for SMT (including via MC groups_flags). On machines where asymmetry attaches immediately above SMT, topology may strip that flag and reduce this branch of behavior, but explicit SMT-aware placement still matters.

Quoted 32 lines
Quoted 32 lines

Patch set summary:

- [PATCH 1/4] sched/fair: Prefer fully-idle SMT cores in asym-capacity idle selection

Prefer fully-idle SMT cores in asym-capacity idle selection. In the wakeup fast path, extend select_idle_capacity() / asym_fits_cpu() so idle selection can prefer CPUs on fully idle cores, with a safe fallback.

- [PATCH 2/4] sched/fair: Reject misfit pulls onto busy SMT siblings on asym-capacity

Reject misfit pulls onto busy SMT siblings on SD_ASYM_CPUCAPACITY. Provided for consistency with PATCH 1/4.

- [PATCH 3/4] sched/fair: Enable EAS with SMT on SD_ASYM_CPUCAPACITY systems

Enable EAS with SD_ASYM_CPUCAPACITY and SMT. Also provided for consistency with PATCH 1/4. I've also tested with/without /proc/sys/kernel/sched_energy_aware enabled (same platform) and haven't noticed any regression.

- [PATCH 4/4] sched/fair: Prefer fully-idle SMT core for NOHZ idle load balancer

When choosing the housekeeping CPU that runs the idle load balancer, prefer an idle CPU on a fully idle core so migrated work lands where effective capacity is available.

The change is still consistent with the same "avoid CPUs with busy sibling" logic and it shows some benefits on Vera, but could have negative impact on other systems, I'm including it for completeness (feedback is appreciated).

Quoted 7 lines
Quoted 4 lines

This patch set has been tested on the new NVIDIA Vera Rubin platform, where SMT is enabled and the firmware exposes small frequency variations (+/-~5%) as differences in CPU capacity, resulting in SD_ASYM_CPUCAPACITY being set.

Are you referring to nominal_freq?

Correct.

Thanks,
-Andrea

#31 lore raw
Quoted 27 lines · Andrea Righi

Hi Balbir,

Quoted 24 lines · Balbir Singh
Quoted 10 lines · Andrea Righi

This series attempts to improve SD_ASYM_CPUCAPACITY scheduling by introducing SMT awareness.

= Problem =

Nominal per-logical-CPU capacity can overstate usable compute when an SMT sibling is busy, because the physical core doesn't deliver its full nominal capacity. So, several SD_ASYM_CPUCAPACITY paths may pick high capacity CPUs that are not actually good destinations.

Quoted 6 lines

= Proposed Solution =

This patch set aligns those paths with a simple rule already used
elsewhere: when SMT is active, prefer fully idle cores and avoid treating partially idle SMT siblings as full-capacity targets where that would mislead load balance.

In kernel/sched/topology.c

	/* Don't attempt to spread across CPUs of different capacities. */
	if ((sd->flags & SD_ASYM_CPUCAPACITY) && sd->child)
		sd->child->flags &= ~SD_PREFER_SIBLING;
Quoted 8 lines
Quoted 1 line

Should handle the selection, but I guess this does not work for SMT level sd's?

IIUC, SD_PREFER_SIBLING steers load balance toward sibling_imbalance() (spread runnables across child/sibling domains), it doesn't encode the fully-idle core first logic. In practice it doesn't give us SMT-aware destination choice when a sibling is busy and this series is trying to cover that gap in the palcement path.

Thanks, so we care about idle selection, not necessarily balancing and yes I did see that sd->child needs to be set for SD_PEFER_SIBLING to be cleared.

Quoted 55 lines

BTW, on Vera the hierarchy is SMT -> MC -> NUMA:

root@localhost:~# grep . /sys/kernel/debug/sched/domains/cpu0/domain*/flags
/sys/kernel/debug/sched/domains/cpu0/domain0/flags:SD_BALANCE_NEWIDLE SD_BALANCE_EXEC SD_BALANCE_FORK SD_WAKE_AFFINE SD_SHARE_CPUCAPACITY SD_SHARE_LLC SD_PREFER_SIBLING
/sys/kernel/debug/sched/domains/cpu0/domain1/flags:SD_BALANCE_NEWIDLE SD_BALANCE_EXEC SD_BALANCE_FORK SD_WAKE_AFFINE SD_ASYM_CPUCAPACITY SD_SHARE_LLC
/sys/kernel/debug/sched/domains/cpu0/domain2/flags:SD_BALANCE_NEWIDLE SD_ASYM_CPUCAPACITY SD_ASYM_CPUCAPACITY_FULL SD_SERIALIZE SD_NUMA

And domain1/groups_flags (child / SMT flags on the sched groups used at the MC level) still has SD_PREFER_SIBLING together with SD_SHARE_CPUCAPACITY.

root@localhost:~# cat /sys/kernel/debug/sched/domains/cpu0/domain1/groups_flags
SD_BALANCE_NEWIDLE SD_BALANCE_EXEC SD_BALANCE_FORK SD_WAKE_AFFINE SD_SHARE_CPUCAPACITY SD_SHARE_LLC SD_PREFER_SIBLING

So, prefer-sibling is still in play for SMT (including via MC groups_flags). On machines where asymmetry attaches immediately above SMT, topology may strip that flag and reduce this branch of behavior, but explicit SMT-aware placement still matters.

Quoted 37 lines
Quoted 36 lines

Patch set summary:

- [PATCH 1/4] sched/fair: Prefer fully-idle SMT cores in asym-capacity idle selection

Prefer fully-idle SMT cores in asym-capacity idle selection. In the wakeup fast path, extend select_idle_capacity() / asym_fits_cpu() so idle selection can prefer CPUs on fully idle cores, with a safe fallback.

- [PATCH 2/4] sched/fair: Reject misfit pulls onto busy SMT siblings on asym-capacity

Reject misfit pulls onto busy SMT siblings on SD_ASYM_CPUCAPACITY. Provided for consistency with PATCH 1/4.

- [PATCH 3/4] sched/fair: Enable EAS with SMT on SD_ASYM_CPUCAPACITY systems

Enable EAS with SD_ASYM_CPUCAPACITY and SMT. Also provided for consistency with PATCH 1/4. I've also tested with/without /proc/sys/kernel/sched_energy_aware enabled (same platform) and haven't noticed any regression.

- [PATCH 4/4] sched/fair: Prefer fully-idle SMT core for NOHZ idle load balancer

When choosing the housekeeping CPU that runs the idle load balancer, prefer an idle CPU on a fully idle core so migrated work lands where effective capacity is available.

The change is still consistent with the same "avoid CPUs with busy sibling" logic and it shows some benefits on Vera, but could have negative impact on other systems, I'm including it for completeness (feedback is appreciated).

This patch set has been tested on the new NVIDIA Vera Rubin platform, where SMT is enabled and the firmware exposes small frequency variations (+/-~5%) as differences in CPU capacity, resulting in SD_ASYM_CPUCAPACITY being set.

Quoted 5 lines
Quoted 2 lines

Are you referring to nominal_freq?

Correct.

Thanks,
Balbir

#32 lore raw
In reply to Andrea Righi

Hi Andrea,

On 26.03.26 16:02, Andrea Righi wrote:

[...]

Quoted 13 lines

This patch set has been tested on the new NVIDIA Vera Rubin platform, where SMT is enabled and the firmware exposes small frequency variations (+/-~5%) as differences in CPU capacity, resulting in SD_ASYM_CPUCAPACITY being set.

Without these patches, performance can drop up to ~2x with CPU-intensive workloads, because the SD_ASYM_CPUCAPACITY idle selection policy does not account for busy SMT siblings.

Alternative approaches have been evaluated, such as equalizing CPU capacities, either by exposing uniform values via firmware (ACPI/CPPC) or normalizing them in the kernel by grouping CPUs within a small capacity window (+-5%) [1][2], or enabling asympacking [3].

Quoted 4 lines

However, adding SMT awareness to SD_ASYM_CPUCAPACITY has shown better results so far. Improving this policy also seems worthwhile in general, as other platforms in the future may enable SMT with asymmetric CPU topologies.

I still wonder whether we really need select_idle_capacity() (plus the smt part) for asymmetric CPU capacity systems where the CPU capacity differences are < 5% of SCHED_CAPACITY_SCALE.

The known example would be the NVIDIA Grace (!smt) server with its slightly different perf_caps.highest_perf values.

We did run DCPerf Mediawiki on this thing with:

(1) ASYM_CPUCAPACITY (default)

(2) NO ASYM_CPUCAPACITY

We also ran on a comparable ARM64 server (!smt) for comparison:

(1) ASYM_CPUCAPACITY

(2) NO ASYM_CPUCAPACITY (default)

Both systems have 72 CPUs, run v6.8 and have a single MC sched domain with LLC spanning over all 72 CPUs. During the tests there were ~750 tasks among them the workload related:

#hhvmworker 147
#mariadbd 204
#memcached 11
#nginx 8
#wrk 144
#ProxygenWorker 1

load_balance:

not_idle 3x more on (2)

idle 2x more on (2)

newly_idle 2-10x more on (2)

wakeup:

move_affine 2-3x more on (1)

ttwu_local 1.5-2 more on (2)

We also instrumented all the bailout conditions in select_task_sibling() (sis())->select_idle_cpu() and select_idle_capacity() (sic()).

In (1) almost all wakeups end up in select_idle_cpu() returning -1 due to the fact that 'sd->shared->nr_idle_scan' under SIS_UTIL is 0. So sis() in (1) almost always returns target (this_cpu or prev_cpu). sic() doesn't do this.

What I haven't done is to try (1) with SIS_UTIL or (2) with NO_SIS_UTIL.

I wonder whether this is the underlying reason for the benefit of (1) over (2) we see here with smt now?

So IMHO before adding smt support to (1) for these small CPPC based CPU capacity differences we should make sure that the same can't be achieved by disabling SIS_UTIL or to soften it a bit.

So does (2) with NO_SIS_UTIL performs worse than (1) with your smt related add-ons in sic()?

#33 lore raw

Hi Dietmar,

Quoted 80 lines · Dietmar Eggemann

Hi Andrea,

On 26.03.26 16:02, Andrea Righi wrote:

[...]

Quoted 13 lines

This patch set has been tested on the new NVIDIA Vera Rubin platform, where SMT is enabled and the firmware exposes small frequency variations (+/-~5%) as differences in CPU capacity, resulting in SD_ASYM_CPUCAPACITY being set.

Without these patches, performance can drop up to ~2x with CPU-intensive workloads, because the SD_ASYM_CPUCAPACITY idle selection policy does not account for busy SMT siblings.

Alternative approaches have been evaluated, such as equalizing CPU capacities, either by exposing uniform values via firmware (ACPI/CPPC) or normalizing them in the kernel by grouping CPUs within a small capacity window (+-5%) [1][2], or enabling asympacking [3].

Quoted 4 lines

However, adding SMT awareness to SD_ASYM_CPUCAPACITY has shown better results so far. Improving this policy also seems worthwhile in general, as other platforms in the future may enable SMT with asymmetric CPU topologies.

I still wonder whether we really need select_idle_capacity() (plus the smt part) for asymmetric CPU capacity systems where the CPU capacity differences are < 5% of SCHED_CAPACITY_SCALE.

The known example would be the NVIDIA Grace (!smt) server with its slightly different perf_caps.highest_perf values.

We did run DCPerf Mediawiki on this thing with:

(1) ASYM_CPUCAPACITY (default)

(2) NO ASYM_CPUCAPACITY

We also ran on a comparable ARM64 server (!smt) for comparison:

(1) ASYM_CPUCAPACITY

(2) NO ASYM_CPUCAPACITY (default)

Both systems have 72 CPUs, run v6.8 and have a single MC sched domain with LLC spanning over all 72 CPUs. During the tests there were ~750 tasks among them the workload related:

#hhvmworker 147
#mariadbd 204
#memcached 11
#nginx 8
#wrk 144
#ProxygenWorker 1

load_balance:

not_idle 3x more on (2)

idle 2x more on (2)

newly_idle 2-10x more on (2)

wakeup:

move_affine 2-3x more on (1)

ttwu_local 1.5-2 more on (2)

We also instrumented all the bailout conditions in select_task_sibling() (sis())->select_idle_cpu() and select_idle_capacity() (sic()).

In (1) almost all wakeups end up in select_idle_cpu() returning -1 due to the fact that 'sd->shared->nr_idle_scan' under SIS_UTIL is 0. So sis() in (1) almost always returns target (this_cpu or prev_cpu). sic() doesn't do this.

What I haven't done is to try (1) with SIS_UTIL or (2) with NO_SIS_UTIL.

I wonder whether this is the underlying reason for the benefit of (1) over (2) we see here with smt now?

Quoted 6 lines

So IMHO before adding smt support to (1) for these small CPPC based CPU capacity differences we should make sure that the same can't be achieved by disabling SIS_UTIL or to soften it a bit.

So does (2) with NO_SIS_UTIL performs worse than (1) with your smt related add-ons in sic()?

Thanks for running these experiments and sharing the data, this is very useful!

I did a quick test on Vera using the NVBLAS benchmark, comparing NO ASYM_CPUCAPACITY with and without SIS_UTIL, but the difference seems to be within error range. I'll also run DCPerf MediaWiki with all the different configurations to see if I get similar results.

More in general, I agree that for small capacity differences (e.g., within ~5%) the benefits of using ASYM_CPUCAPACITY is questionable. And I'm also fine to go back to the idea of grouping together CPUS within the 5% capacity window, if we think it's a safer approach (results in your case are quite evident, and BTW, that means we also shouldn't have ASYM_CPU_CAPACITY on Grace, so in theory the 5% threshold should also improve performance on Grace, that doesn't have SMT).

That said, I still think there's value in adding SMT awareness to select_idle_capacity(). Even if we decide to avoid ASYM_CPUCAPACITY for small capacity deltas, we should ensure that the behavior remains reasonable if both features are enabled, for any reason. Right now, there are cases where the current behavior leads to significant performance degradation (~2x), so having a mechanism to prevent clearly suboptimal task placement still seems worthwhile. Essentially, what I'm saying is that one thing doesn't exclude the other.

Thanks,
-Andrea

#34 lore raw
Quoted 6 lines · Andrea Righi

Hi Dietmar,

Quoted 3 lines · Dietmar Eggemann

Hi Andrea,

On 26.03.26 16:02, Andrea Righi wrote:

[...]

Quoted 9 lines
Quoted 2 lines

So does (2) with NO_SIS_UTIL performs worse than (1) with your smt related add-ons in sic()?

Thanks for running these experiments and sharing the data, this is very useful!

I did a quick test on Vera using the NVBLAS benchmark, comparing NO ASYM_CPUCAPACITY with and without SIS_UTIL, but the difference seems to be within error range. I'll also run DCPerf MediaWiki with all the different

I'm not familiar with the NVBLAS benchmark. Does it drive your system into 'sd->shared->nr_idle_scan = 0' state?

We just have to understand where this benefit of using sic() instead of sis() is coming from. I'm doubtful that this is the best_cpu thing after if (!choose_idle_cpu(cpu, p)) in sic()'s for_each_cpu_wrap(cpu, cpus, target) loop given that the CPU capacity diffs are so small.

Quoted 9 lines

configurations to see if I get similar results.

More in general, I agree that for small capacity differences (e.g., within ~5%) the benefits of using ASYM_CPUCAPACITY is questionable. And I'm also fine to go back to the idea of grouping together CPUS within the 5% capacity window, if we think it's a safer approach (results in your case are quite evident, and BTW, that means we also shouldn't have ASYM_CPU_CAPACITY on Grace, so in theory the 5% threshold should also improve performance on Grace, that doesn't have SMT).

There shouldn't be so many machines with these binning-introduced small CPU capacity diffs out there? In fact, I only know about your Grace (!smt) and Vera (smt) machines.

Quoted 8 lines

That said, I still think there's value in adding SMT awareness to select_idle_capacity(). Even if we decide to avoid ASYM_CPUCAPACITY for small capacity deltas, we should ensure that the behavior remains reasonable if both features are enabled, for any reason. Right now, there are cases where the current behavior leads to significant performance degradation (~2x), so having a mechanism to prevent clearly suboptimal task placement still seems worthwhile. Essentially, what I'm saying is that one thing doesn't exclude the other.

IMHO, in case we would know where this improvement is coming from using sic() instead of default sis() (which already as smt support) then maybe, it's a lot of extra code at the end ... And mobile big.LITTLE (with larger CPU capacity diffs) doesn't have smt.

maintainer
#35 lore raw
Quoted 39 lines · Dietmar Eggemann
Quoted 6 lines · Andrea Righi

Hi Dietmar,

Quoted 3 lines · Dietmar Eggemann

Hi Andrea,

On 26.03.26 16:02, Andrea Righi wrote:

[...]

Quoted 9 lines
Quoted 2 lines

So does (2) with NO_SIS_UTIL performs worse than (1) with your smt related add-ons in sic()?

Thanks for running these experiments and sharing the data, this is very useful!

I did a quick test on Vera using the NVBLAS benchmark, comparing NO ASYM_CPUCAPACITY with and without SIS_UTIL, but the difference seems to be within error range. I'll also run DCPerf MediaWiki with all the different

I'm not familiar with the NVBLAS benchmark. Does it drive your system into 'sd->shared->nr_idle_scan = 0' state?

We just have to understand where this benefit of using sic() instead of sis() is coming from. I'm doubtful that this is the best_cpu thing after if (!choose_idle_cpu(cpu, p)) in sic()'s for_each_cpu_wrap(cpu, cpus, target) loop given that the CPU capacity diffs are so small.

Quoted 9 lines

configurations to see if I get similar results.

More in general, I agree that for small capacity differences (e.g., within ~5%) the benefits of using ASYM_CPUCAPACITY is questionable. And I'm also fine to go back to the idea of grouping together CPUS within the 5% capacity window, if we think it's a safer approach (results in your case are quite evident, and BTW, that means we also shouldn't have ASYM_CPU_CAPACITY on Grace, so in theory the 5% threshold should also improve performance on Grace, that doesn't have SMT).

Quoted 3 lines

There shouldn't be so many machines with these binning-introduced small CPU capacity diffs out there? In fact, I only know about your Grace (!smt) and Vera (smt) machines.

In any case it's always better to add the support than enabling asym_packing

Quoted 10 lines
Quoted 9 lines

That said, I still think there's value in adding SMT awareness to select_idle_capacity(). Even if we decide to avoid ASYM_CPUCAPACITY for small capacity deltas, we should ensure that the behavior remains reasonable if both features are enabled, for any reason. Right now, there are cases where the current behavior leads to significant performance degradation (~2x), so having a mechanism to prevent clearly suboptimal task placement still seems worthwhile. Essentially, what I'm saying is that one thing doesn't exclude the other.

Quoted 4 lines

IMHO, in case we would know where this improvement is coming from using sic() instead of default sis() (which already as smt support) then maybe, it's a lot of extra code at the end ... And mobile big.LITTLE (with larger CPU capacity diffs) doesn't have smt.

The last proposal based on prateek proposal in sic() doesn't seems that large

#36 lore raw
Quoted 18 lines · Vincent Guittot
Quoted 17 lines · Dietmar Eggemann
Quoted 6 lines · Andrea Righi

Hi Dietmar,

Quoted 3 lines · Dietmar Eggemann

Hi Andrea,

On 26.03.26 16:02, Andrea Righi wrote:

[...]

Quoted 6 lines
Quoted 2 lines

So does (2) with NO_SIS_UTIL performs worse than (1) with your smt related add-ons in sic()?

Thanks for running these experiments and sharing the data, this is very useful!

Quoted 6 lines
Quoted 6 lines
Quoted 3 lines

I did a quick test on Vera using the NVBLAS benchmark, comparing NO ASYM_CPUCAPACITY with and without SIS_UTIL, but the difference seems to be within error range. I'll also run DCPerf MediaWiki with all the different

I'm not familiar with the NVBLAS benchmark. Does it drive your system into 'sd->shared->nr_idle_scan = 0' state?

It's something internally unfortunately... it's just running a single CPU-intensive task for each SMT core (in practice half of the CPUs tasks). I don't think we're hitting sd->shared->nr_idle_scan == 0 in this case.

Quoted 32 lines
Quoted 16 lines

We just have to understand where this benefit of using sic() instead of sis() is coming from. I'm doubtful that this is the best_cpu thing after if (!choose_idle_cpu(cpu, p)) in sic()'s for_each_cpu_wrap(cpu, cpus, target) loop given that the CPU capacity diffs are so small.

Quoted 9 lines

configurations to see if I get similar results.

More in general, I agree that for small capacity differences (e.g., within ~5%) the benefits of using ASYM_CPUCAPACITY is questionable. And I'm also fine to go back to the idea of grouping together CPUS within the 5% capacity window, if we think it's a safer approach (results in your case are quite evident, and BTW, that means we also shouldn't have ASYM_CPU_CAPACITY on Grace, so in theory the 5% threshold should also improve performance on Grace, that doesn't have SMT).

Quoted 3 lines

There shouldn't be so many machines with these binning-introduced small CPU capacity diffs out there? In fact, I only know about your Grace (!smt) and Vera (smt) machines.

In any case it's always better to add the support than enabling asym_packing

Quoted 10 lines
Quoted 9 lines

That said, I still think there's value in adding SMT awareness to select_idle_capacity(). Even if we decide to avoid ASYM_CPUCAPACITY for small capacity deltas, we should ensure that the behavior remains reasonable if both features are enabled, for any reason. Right now, there are cases where the current behavior leads to significant performance degradation (~2x), so having a mechanism to prevent clearly suboptimal task placement still seems worthwhile. Essentially, what I'm saying is that one thing doesn't exclude the other.

Quoted 6 lines
Quoted 4 lines

IMHO, in case we would know where this improvement is coming from using sic() instead of default sis() (which already as smt support) then maybe, it's a lot of extra code at the end ... And mobile big.LITTLE (with larger CPU capacity diffs) doesn't have smt.

The last proposal based on prateek proposal in sic() doesn't seems that large

Exactly, I was referring just to that patch, which would solve the big part of the performance issue. We can ignore the ILB part for now.

Thanks,
-Andrea

#37 lore raw
Quoted 23 lines · Andrea Righi
Quoted 22 lines · Vincent Guittot
Quoted 21 lines · Dietmar Eggemann
Quoted 6 lines · Andrea Righi

Hi Dietmar,

Quoted 3 lines · Dietmar Eggemann

Hi Andrea,

On 26.03.26 16:02, Andrea Righi wrote:

[...]

Quoted 9 lines
Quoted 2 lines

So does (2) with NO_SIS_UTIL performs worse than (1) with your smt related add-ons in sic()?

Thanks for running these experiments and sharing the data, this is very useful!

I did a quick test on Vera using the NVBLAS benchmark, comparing NO ASYM_CPUCAPACITY with and without SIS_UTIL, but the difference seems to be within error range. I'll also run DCPerf MediaWiki with all the different

Quoted 6 lines
Quoted 2 lines
Quoted 2 lines

I'm not familiar with the NVBLAS benchmark. Does it drive your system into 'sd->shared->nr_idle_scan = 0' state?

It's something internally unfortunately... it's just running a single CPU-intensive task for each SMT core (in practice half of the CPUs tasks). I don't think we're hitting sd->shared->nr_idle_scan == 0 in this case.

Just finished running some tests with DCPerf MediaWiki on Vera as well (sorry, it took a while, I did mutliple runs to rule out potential flukes):

 +---------------------------------+--------+--------+--------+--------+
 | Configuration                   |   rps  |  p50   |  p95   |  p99   |
 +---------------------------------+--------+--------+--------+--------+
 | NO ASYM + SIS_UTIL              |  8113  |  0.067 |  0.184 |  0.225 |
 | NO ASYM + NO_SIS_UTIL           |  8093  |  0.068 |  0.184 |  0.223 |
 |                                 |        |        |        |        |
 | ASYM + SMT + SIS_UTIL           |  8129  |  0.076 |  0.149 |  0.188 |
 | ASYM + SMT + NO_SIS_UTIL        |  8138  |  0.076 |  0.148 |  0.186 |
 |                                 |        |        |        |        |
 | ASYM + ILB SMT + SIS_UTIL       |  8189  |  0.075 |  0.150 |  0.189 |
 | ASYM + SMT + ILB SMT + SIS_UTIL |  8185  |  0.076 |  0.151 |  0.190 |
 +---------------------------------+--------+--------+--------+--------+

Looking at the data:
- SIS_UTIL doesn't seem relevant in this case (differences are within error range),
- ASYM_CPU_CAPACITY seems to provide a small throughput gain, but it seems more beneficial for tail latency reduction,
- the ILB SMT patch seems to slightly improve throughput, but the biggest benefit is still coming from ASYM_CPU_CAPACITY.

Overall, also in this case it seems beneficial to use ASYM_CPU_CAPACITY rather than equalizing the capacities.

That said, I'm still not sure why ASYM is helping. The frequency asymmetry is really small (~2%), so the latency improvements are unlikely to come from prioritizing the faster cores, as that should mainly affect throughput rather than tail latency and likely to a smaller extent.

Thanks,
-Andrea

#38 lore raw
Quoted 11 lines · Andrea Righi
Quoted 10 lines · Andrea Righi
Quoted 9 lines · Vincent Guittot
Quoted 8 lines · Dietmar Eggemann
Quoted 6 lines · Andrea Righi

Hi Dietmar,

Quoted 3 lines · Dietmar Eggemann

Hi Andrea,

On 26.03.26 16:02, Andrea Righi wrote:

[...]

Quoted 5 lines

Just finished running some tests with DCPerf MediaWiki on Vera as well (sorry, it took a while, I did mutliple runs to rule out potential flukes):

 +---------------------------------+--------+--------+--------+--------+
 | Configuration                   |   rps  |  p50   |  p95   |  p99   |

Just to make sure: rps -> "Wrk RPS" and pXX -> "Nginx PXX time" in run_details.json ?

Quoted 3 lines
 +---------------------------------+--------+--------+--------+--------+
 | NO ASYM + SIS_UTIL              |  8113  |  0.067 |  0.184 |  0.225 |
 | NO ASYM + NO_SIS_UTIL           |  8093  |  0.068 |  0.184 |  0.223 |

Thanks for the test results! Ok, so SIS_UTIL doesn't seem to play a role here. This workload should have #runnable tasks > #CPUs.

Still trying to grasp why 'sic() + smt' is better than 'sis() + smt' for NVBLAS?

There is a subtle differences in the start cpu for iterating:

sis(): for_each_cpu_wrap(cpu, cpus, target + 1)

                                           ^^^

sic(): for_each_cpu_wrap(cpu, cpus, target)

Not sure if this makes all the difference?

Quoted 3 lines
 |                                 |        |        |        |        |
 | ASYM + SMT + SIS_UTIL           |  8129  |  0.076 |  0.149 |  0.188 |
 | ASYM + SMT + NO_SIS_UTIL        |  8138  |  0.076 |  0.148 |  0.186 |

This should be the same, right? SIS_UTIL is only for sis() so when using sic() this shouldn't differ. Or did you code SIS_UTIL into sic()?

Quoted 4 lines
 |                                 |        |        |        |        |
 | ASYM + ILB SMT + SIS_UTIL       |  8189  |  0.075 |  0.150 |  0.189 |
 | ASYM + SMT + ILB SMT + SIS_UTIL |  8185  |  0.076 |  0.151 |  0.190 |
 +---------------------------------+--------+--------+--------+--------+

So with '#tasks > #CPUs' smt doesn't make a difference.

Quoted 8 lines

Looking at the data:
- SIS_UTIL doesn't seem relevant in this case (differences are within error range),
- ASYM_CPU_CAPACITY seems to provide a small throughput gain, but it seems more beneficial for tail latency reduction,
- the ILB SMT patch seems to slightly improve throughput, but the biggest benefit is still coming from ASYM_CPU_CAPACITY.

Quoted 4 lines

Overall, also in this case it seems beneficial to use ASYM_CPU_CAPACITY rather than equalizing the capacities.

That said, I'm still not sure why ASYM is helping. The frequency asymmetry

OK, I still would be more comfortable with this when I would now why this is :-)

Quoted 3 lines

is really small (~2%), so the latency improvements are unlikely to come from prioritizing the faster cores, as that should mainly affect throughput rather than tail latency and likely to a smaller extent.

[...]

#39 lore raw

Hi Dietmar,

Quoted 15 lines · Dietmar Eggemann
Quoted 11 lines · Andrea Righi
Quoted 10 lines · Andrea Righi
Quoted 9 lines · Vincent Guittot
Quoted 8 lines · Dietmar Eggemann
Quoted 6 lines · Andrea Righi

Hi Dietmar,

Quoted 3 lines · Dietmar Eggemann

Hi Andrea,

On 26.03.26 16:02, Andrea Righi wrote:

[...]

Quoted 8 lines
Quoted 5 lines

Just finished running some tests with DCPerf MediaWiki on Vera as well (sorry, it took a while, I did mutliple runs to rule out potential flukes):

 +---------------------------------+--------+--------+--------+--------+
 | Configuration                   |   rps  |  p50   |  p95   |  p99   |

Just to make sure: rps -> "Wrk RPS" and pXX -> "Nginx PXX time" in run_details.json ?

Correct, rps == "Wrk RPS", p50 == "Nginx P50 time", etc.

Quoted 10 lines
Quoted 4 lines
 +---------------------------------+--------+--------+--------+--------+
 | NO ASYM + SIS_UTIL              |  8113  |  0.067 |  0.184 |  0.225 |
 | NO ASYM + NO_SIS_UTIL           |  8093  |  0.068 |  0.184 |  0.223 |

Thanks for the test results! Ok, so SIS_UTIL doesn't seem to play a role here. This workload should have #runnable tasks > #CPUs.

Still trying to grasp why 'sic() + smt' is better than 'sis() + smt' for NVBLAS?

Same...

Quoted 8 lines

There is a subtle differences in the start cpu for iterating:

sis(): for_each_cpu_wrap(cpu, cpus, target + 1)

                                           ^^^

sic(): for_each_cpu_wrap(cpu, cpus, target)

Not sure if this makes all the difference?

I quickly matching the wrap start (both ways), but still doesn't make any
difference: sic() is still slightly better than sis(). So the performance gap doesn't seem to be in the wrap origin.

Quoted 7 lines
Quoted 4 lines
 |                                 |        |        |        |        |
 | ASYM + SMT + SIS_UTIL           |  8129  |  0.076 |  0.149 |  0.188 |
 | ASYM + SMT + NO_SIS_UTIL        |  8138  |  0.076 |  0.148 |  0.186 |

This should be the same, right? SIS_UTIL is only for sis() so when using sic() this shouldn't differ. Or did you code SIS_UTIL into sic()?

No, you're right, it should be the same, SIS_UTIL is irrelevant here.

Quoted 7 lines
Quoted 5 lines
 |                                 |        |        |        |        |
 | ASYM + ILB SMT + SIS_UTIL       |  8189  |  0.075 |  0.150 |  0.189 |
 | ASYM + SMT + ILB SMT + SIS_UTIL |  8185  |  0.076 |  0.151 |  0.190 |
 +---------------------------------+--------+--------+--------+--------+

So with '#tasks > #CPUs' smt doesn't make a difference.

Correct. At saturation there's no benefit with the SMT awareness, which makes sense, all CPUs/siblings are busy, so there's no preferred fully-idle SMT core to prioritize.

Quoted 9 lines
Quoted 8 lines

Looking at the data:
- SIS_UTIL doesn't seem relevant in this case (differences are within error range),
- ASYM_CPU_CAPACITY seems to provide a small throughput gain, but it seems more beneficial for tail latency reduction,
- the ILB SMT patch seems to slightly improve throughput, but the biggest benefit is still coming from ASYM_CPU_CAPACITY.

Quoted 7 lines
Quoted 4 lines

Overall, also in this case it seems beneficial to use ASYM_CPU_CAPACITY rather than equalizing the capacities.

That said, I'm still not sure why ASYM is helping. The frequency asymmetry

OK, I still would be more comfortable with this when I would now why this is :-)

Working on this. :)

Thanks,
-Andrea

#40 lore raw
Quoted 1 line · Andrea Righi

On Fri, Apr 03, 2026 at 01:47:17PM +0200, Dietmar Eggemann wrote:

...

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

Looking at the data:
- SIS_UTIL doesn't seem relevant in this case (differences are within error range),
- ASYM_CPU_CAPACITY seems to provide a small throughput gain, but it seems more beneficial for tail latency reduction,
- the ILB SMT patch seems to slightly improve throughput, but the biggest benefit is still coming from ASYM_CPU_CAPACITY.

Overall, also in this case it seems beneficial to use ASYM_CPU_CAPACITY rather than equalizing the capacities.

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

That said, I'm still not sure why ASYM is helping. The frequency asymmetry

OK, I still would be more comfortable with this when I would now why this is :-)

Working on this. :)

Alright, I think I found something. I tried to make sis() behave more like sic() by adding the same SMT "full idle core" check in the fast path and removing the extra select_idle_smt(prev) hop from the LLC idle path.

Essentially this:

kernel/sched/fair.c +2 -37
index 7bebceb5ed9df..19fffa2df2d36 100644
@@ -7651,29 +7651,6 @@ static int select_idle_core(struct task_struct *p, int core, struct cpumask *cpu
 	return -1;
 }
 
-/*
- * Scan the local SMT mask for idle CPUs.
- */
-static int select_idle_smt(struct task_struct *p, struct sched_domain *sd, int target)
-{
-	int cpu;
-
-	for_each_cpu_and(cpu, cpu_smt_mask(target), p->cpus_ptr) {
-		if (cpu == target)
-			continue;
-		/*
-		 * Check if the CPU is in the LLC scheduling domain of @target.
-		 * Due to isolcpus, there is no guarantee that all the siblings are in the domain.
-		 */
-		if (!cpumask_test_cpu(cpu, sched_domain_span(sd)))
-			continue;
-		if (available_idle_cpu(cpu) || sched_idle_cpu(cpu))
-			return cpu;
-	}
-
-	return -1;
-}
-
 #else /* !CONFIG_SCHED_SMT: */
 
 static inline void set_idle_cores(int cpu, int val)
@@ -7690,11 +7667,6 @@ static inline int select_idle_core(struct task_struct *p, int core, struct cpuma
 	return __select_idle_cpu(core, p);
 }
 
-static inline int select_idle_smt(struct task_struct *p, struct sched_domain *sd, int target)
-{
-	return -1;
-}
-
 #endif /* !CONFIG_SCHED_SMT */
 
 /*
@@ -7859,7 +7831,7 @@ static inline bool asym_fits_cpu(unsigned long util,
 		       (util_fits_cpu(util, util_min, util_max, cpu) > 0);
 	}
 
-	return true;
+	return !sched_smt_active() || is_core_idle(cpu);
 }
 
 /*
@@ -7964,16 +7936,9 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
 	if (!sd)
 		return target;
 
-	if (sched_smt_active()) {
+	if (sched_smt_active())
 		has_idle_core = test_idle_cores(target);
 
-		if (!has_idle_core && cpus_share_cache(prev, target)) {
-			i = select_idle_smt(p, sd, prev);
-			if ((unsigned int)i < nr_cpumask_bits)
-				return i;
-		}
-	}
-
 	i = select_idle_cpu(p, sd, has_idle_core, target);
 	if ((unsigned)i < nr_cpumask_bits)
 		return i;

---

With this applied, I see identical performance between NO_ASYM and ASYM+SMT.

I'm not suggesting to apply this, but that seems to be the reason why ASYM+SMT performs better in my case.

-Andrea

#41 lore raw
Quoted 43 lines · Andrea Righi
Quoted 1 line · Andrea Righi

On Fri, Apr 03, 2026 at 01:47:17PM +0200, Dietmar Eggemann wrote:

...

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

Looking at the data:
- SIS_UTIL doesn't seem relevant in this case (differences are within error range),
- ASYM_CPU_CAPACITY seems to provide a small throughput gain, but it seems more beneficial for tail latency reduction,
- the ILB SMT patch seems to slightly improve throughput, but the biggest benefit is still coming from ASYM_CPU_CAPACITY.

Overall, also in this case it seems beneficial to use ASYM_CPU_CAPACITY rather than equalizing the capacities.

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

That said, I'm still not sure why ASYM is helping. The frequency asymmetry

OK, I still would be more comfortable with this when I would now why this is :-)

Working on this. :)

Alright, I think I found something. I tried to make sis() behave more like sic() by adding the same SMT "full idle core" check in the fast path and removing the extra select_idle_smt(prev) hop from the LLC idle path.

Essentially this:

kernel/sched/fair.c +0 -8
index 7bebceb5ed9df..19fffa2df2d36 100644
@@ -7651,29 +7651,6 @@ static int select_idle_core(struct task_struct *p, int core, struct cpumask *cpu
 	return -1;
 }
 
-/*
- * Scan the local SMT mask for idle CPUs.
- */
-static int select_idle_smt(struct task_struct *p, struct sched_domain *sd, int target)
-{
-	int cpu;
-
-	for_each_cpu_and(cpu, cpu_smt_mask(target), p->cpus_ptr) {

kernel/sched/fair.c static int select_idle_core(struct task_struct *p, int core, struct cpumask *cpu · line 7671

Quoted 10 lines
kernel/sched/fair.c +0 -10
-		if (cpu == target)
-			continue;
-		/*
-		 * Check if the CPU is in the LLC scheduling domain of @target.
-		 * Due to isolcpus, there is no guarantee that all the siblings are in the domain.
-		 */
-		if (!cpumask_test_cpu(cpu, sched_domain_span(sd)))
-			continue;
-		if (available_idle_cpu(cpu) || sched_idle_cpu(cpu)) 
-			return cpu;

So it is this returning of CPU from the smt mask rather than the

    for_each_cpu_wrap(cpu, cpus, target + 1)

        __select_idle_cpu()

            if (choose_idle_cpu(cpu, p) && ...)
                return cpu

where cpus is cpumask_and(cpus, sched_domain_span(MC), p->cpus_ptr)

I wonder wether this has anything to do with your NVIDIA Spatial Multithreading (SMT) versus Traditional (time-shared resources) SMT?

Quoted 21 lines
kernel/sched/fair.c +0 -10
-	}
-
-	return -1;
-}
-
 #else /* !CONFIG_SCHED_SMT: */
 
 static inline void set_idle_cores(int cpu, int val)
@@ -7690,11 +7667,6 @@ static inline int select_idle_core(struct task_struct *p, int core, struct cpuma
 	return __select_idle_cpu(core, p);
 }
 
-static inline int select_idle_smt(struct task_struct *p, struct sched_domain *sd, int target)
-{
-	return -1;
-}
-
 #endif /* !CONFIG_SCHED_SMT */
 
 /*

kernel/sched/fair.c static inline bool asym_fits_cpu(unsigned long util, · line 7835

Quoted 6 lines
kernel/sched/fair.c +1 -1
@@ -7859,7 +7831,7 @@ static inline bool asym_fits_cpu(unsigned long util,
 		       (util_fits_cpu(util, util_min, util_max, cpu) > 0);
 	}
 
-	return true;
+	return !sched_smt_active() || is_core_idle(cpu);
 }

This change seems to be orthogonal to the removal of select_idle_smt() for sis()?

BTW, the is_core_idle() in asym_fits_cpu() (used for those early return CPU conditions in sis()) is something we don't have on the NO_ASYM side where we only use choose_idle_cpu().

Quoted 22 lines
kernel/sched/fair.c +1 -9
 /*
@@ -7964,16 +7936,9 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
 	if (!sd)
 		return target;
 
-	if (sched_smt_active()) {
+	if (sched_smt_active())
 		has_idle_core = test_idle_cores(target);
 
-		if (!has_idle_core && cpus_share_cache(prev, target)) {
-			i = select_idle_smt(p, sd, prev);
-			if ((unsigned int)i < nr_cpumask_bits)
-				return i;
-		}
-	}
-
 	i = select_idle_cpu(p, sd, has_idle_core, target);
 	if ((unsigned)i < nr_cpumask_bits)
 		return i;
 
---
Quoted 1 line

With this applied, I see identical performance between NO_ASYM and ASYM+SMT.

Interesting!

Quoted 4 lines

I'm not suggesting to apply this, but that seems to be the reason why ASYM+SMT performs better in my case.

-Andrea

#42 lore raw

Hi Dietmar,

Quoted 57 lines · Dietmar Eggemann
Quoted 43 lines · Andrea Righi
Quoted 1 line · Andrea Righi

On Fri, Apr 03, 2026 at 01:47:17PM +0200, Dietmar Eggemann wrote:

...

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

Looking at the data:
- SIS_UTIL doesn't seem relevant in this case (differences are within error range),
- ASYM_CPU_CAPACITY seems to provide a small throughput gain, but it seems more beneficial for tail latency reduction,
- the ILB SMT patch seems to slightly improve throughput, but the biggest benefit is still coming from ASYM_CPU_CAPACITY.

Overall, also in this case it seems beneficial to use ASYM_CPU_CAPACITY rather than equalizing the capacities.

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

That said, I'm still not sure why ASYM is helping. The frequency asymmetry

OK, I still would be more comfortable with this when I would now why this is :-)

Working on this. :)

Alright, I think I found something. I tried to make sis() behave more like sic() by adding the same SMT "full idle core" check in the fast path and removing the extra select_idle_smt(prev) hop from the LLC idle path.

Essentially this:

kernel/sched/fair.c +0 -8
index 7bebceb5ed9df..19fffa2df2d36 100644
@@ -7651,29 +7651,6 @@ static int select_idle_core(struct task_struct *p, int core, struct cpumask *cpu
 	return -1;
 }
 
-/*
- * Scan the local SMT mask for idle CPUs.
- */
-static int select_idle_smt(struct task_struct *p, struct sched_domain *sd, int target)
-{
-	int cpu;
-
-	for_each_cpu_and(cpu, cpu_smt_mask(target), p->cpus_ptr) {

kernel/sched/fair.c static int select_idle_core(struct task_struct *p, int core, struct cpumask *cpu · line 7671

Quoted 10 lines
kernel/sched/fair.c +0 -10
-		if (cpu == target)
-			continue;
-		/*
-		 * Check if the CPU is in the LLC scheduling domain of @target.
-		 * Due to isolcpus, there is no guarantee that all the siblings are in the domain.
-		 */
-		if (!cpumask_test_cpu(cpu, sched_domain_span(sd)))
-			continue;
-		if (available_idle_cpu(cpu) || sched_idle_cpu(cpu)) 
-			return cpu;

So it is this returning of CPU from the smt mask rather than the

Quoted 8 lines
    for_each_cpu_wrap(cpu, cpus, target + 1)

        __select_idle_cpu()

            if (choose_idle_cpu(cpu, p) && ...)
                return cpu

where cpus is cpumask_and(cpus, sched_domain_span(MC), p->cpus_ptr)

Right, and this is a different behavior that I was trying to eliminate from sis() to make it similar to sic().

Quoted 3 lines

I wonder wether this has anything to do with your NVIDIA Spatial Multithreading (SMT) versus Traditional (time-shared resources) SMT?

I don't have data to prove or disprove that... it'd be interesting to try the same approach on a system with traditional SMT.

Quoted 26 lines
Quoted 26 lines
kernel/sched/fair.c +0 -10
-	}
-
-	return -1;
-}
-
 #else /* !CONFIG_SCHED_SMT: */
 
 static inline void set_idle_cores(int cpu, int val)
@@ -7690,11 +7667,6 @@ static inline int select_idle_core(struct task_struct *p, int core, struct cpuma
 	return __select_idle_cpu(core, p);
 }
 
-static inline int select_idle_smt(struct task_struct *p, struct sched_domain *sd, int target)
-{
-	return -1;
-}
-
 #endif /* !CONFIG_SCHED_SMT */
 
 /*
@@ -7859,7 +7831,7 @@ static inline bool asym_fits_cpu(unsigned long util,
 		       (util_fits_cpu(util, util_min, util_max, cpu) > 0);
 	}
 
Quoted 6 lines

kernel/sched/fair.c static inline bool asym_fits_cpu(unsigned long util, · line 7835

Quoted 3 lines
kernel/sched/fair.c +1 -1
-	return true;
+	return !sched_smt_active() || is_core_idle(cpu);
 }

This change seems to be orthogonal to the removal of select_idle_smt() for sis()?

Right, essentially this modifies sis() to return only if cpu is a fully-idle core.

Quoted 4 lines

BTW, the is_core_idle() in asym_fits_cpu() (used for those early return CPU conditions in sis()) is something we don't have on the NO_ASYM side where we only use choose_idle_cpu().

You mean without this change? In that case, yes, because asym_fits_cpu() was just a no-op. This is one of the behavior changes in sis() to make it similar to sic() with SMT awareness.

Quoted 21 lines
Quoted 21 lines
kernel/sched/fair.c +1 -8
 /*
@@ -7964,16 +7936,9 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
 	if (!sd)
 		return target;
 
-	if (sched_smt_active()) {
+	if (sched_smt_active())
 		has_idle_core = test_idle_cores(target);
 
-		if (!has_idle_core && cpus_share_cache(prev, target)) {
-			i = select_idle_smt(p, sd, prev);
-			if ((unsigned int)i < nr_cpumask_bits)
-				return i;
-		}
-	}
-
 	i = select_idle_cpu(p, sd, has_idle_core, target);
 	if ((unsigned)i < nr_cpumask_bits)
 		return i;
Quoted 11 lines
Quoted 3 lines

---

With this applied, I see identical performance between NO_ASYM and ASYM+SMT.

Interesting!

Quoted 4 lines

I'm not suggesting to apply this, but that seems to be the reason why ASYM+SMT performs better in my case.

-Andrea

Thanks,
-Andrea

#43 lore raw
In reply to Andrea Righi
Quoted 10 lines · Andrea Righi
Quoted 9 lines · Vincent Guittot
Quoted 8 lines · Dietmar Eggemann
Quoted 6 lines · Andrea Righi

Hi Dietmar,

Quoted 3 lines · Dietmar Eggemann

Hi Andrea,

On 26.03.26 16:02, Andrea Righi wrote:

[...]

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

I did a quick test on Vera using the NVBLAS benchmark, comparing NO ASYM_CPUCAPACITY with and without SIS_UTIL, but the difference seems to be within error range. I'll also run DCPerf MediaWiki with all the different

Ah, but this benchmark with '#tasks == #cores' is tailored for this prefer_core thing. And SIS_UTIL shouldn't close the idle CPU search.

Quoted 6 lines
Quoted 2 lines
Quoted 2 lines

I'm not familiar with the NVBLAS benchmark. Does it drive your system into 'sd->shared->nr_idle_scan = 0' state?

It's something internally unfortunately... it's just running a single CPU-intensive task for each SMT core (in practice half of the CPUs tasks). I don't think we're hitting sd->shared->nr_idle_scan == 0 in this case.

OK.

Quoted 15 lines
Quoted 15 lines
Quoted 15 lines

We just have to understand where this benefit of using sic() instead of sis() is coming from. I'm doubtful that this is the best_cpu thing after if (!choose_idle_cpu(cpu, p)) in sic()'s for_each_cpu_wrap(cpu, cpus, target) loop given that the CPU capacity diffs are so small.

Quoted 9 lines

configurations to see if I get similar results.

More in general, I agree that for small capacity differences (e.g., within ~5%) the benefits of using ASYM_CPUCAPACITY is questionable. And I'm also fine to go back to the idea of grouping together CPUS within the 5% capacity window, if we think it's a safer approach (results in your case are quite evident, and BTW, that means we also shouldn't have ASYM_CPU_CAPACITY on Grace, so in theory the 5% threshold should also improve performance on Grace, that doesn't have SMT).

Quoted 5 lines
Quoted 5 lines
Quoted 3 lines

There shouldn't be so many machines with these binning-introduced small CPU capacity diffs out there? In fact, I only know about your Grace (!smt) and Vera (smt) machines.

In any case it's always better to add the support than enabling asym_packing

Yeah, the question for me is more between existing 'sis() + smt' or this new 'sic() + smt' with those minor CPU capacity differences.

[...]

Quoted 9 lines
Quoted 6 lines
Quoted 4 lines

IMHO, in case we would know where this improvement is coming from using sic() instead of default sis() (which already as smt support) then maybe, it's a lot of extra code at the end ... And mobile big.LITTLE (with larger CPU capacity diffs) doesn't have smt.

The last proposal based on prateek proposal in sic() doesn't seems that large

Exactly, I was referring just to that patch, which would solve the big part of the performance issue. We can ignore the ILB part for now.

OK, I see. It's in your v2 you sent out earlier today so I will comment there.