Enable Cluster Scheduling for x86 Hybrid CPUs
cover Enable Cluster Scheduling for x86 Hybrid CPUs
Cluster scheduling domain is not enabled on x86 hybrid CPUs as the logic is missing to do proper load balancing between a cluster with SMT CPUs in single core and a cluster with multiple Atom CPUs.
When cluster scheduling was first introduced to x86, it was noticed that with cluster scheduling on hybrid CPU, single threaded task often ended up on Atom core (or E-core) instead on idle Big core (or P-core), resulting in lower performance. Hence cluster scheduling was disabled on x86 hybrid CPU. (See: https://www.phoronix.com/review/linux-516-regress)
Ricardo recently introduced a patch series that greatly improved the load balancing logic between P-cores and E-cores on x86 hybrid CPUs.
https://lore.kernel.org/lkml/[email protected]/T/#m16ebc8de64dbf4c54adebab701b42b47805105f4
However, that patch series is not enough to allow the enabling of cluster scheduling on hybrid x86 CPUs. This patch series provides some additional fixes needed for load balancing between cluster sched group consisting of SMT CPUs of Big cores and cluster sched group consisting of Atom CPUs. With these patches applied on top of Ricardo's patch series, load is properly balanced between the P-core and E-core clusters. Idle CPUs are used in the proper order:
1) SMT CPU on an idle P-core,
2) idle E-core,
3) unused SMT CPU with a busy sibling.
On x86, CPUs in a cluster share L2 cache. Load is now balanced between the clusters with cluster enabled, for potentially less L2 cache contention.
I did some experiments on an Alder Lake with 6 P-cores and 8 E-cores, organized in two clusters of 4 E-core each.
I tested some single threaded benchmarks in Phoronix suite that previously have shown regressions when cluster scheduling was first enabled. Cluster scheduling using this patch series performs as well as vanilla kernel.
Single Threaded 6.3-rc5 with cluster Improvement
Benchmark scheduling in Performance
(run-run deviation)
-------------------------------------------------------------------------------------------
tjbench (+/- 0.08%) (+/- 0.23%) -0.23%
PhPbench (+/- 0.31%) (+/- 0.89%) -0.39%
flac (+/- 0.58%) (+/- 0.22%) +0.17%
pybench (+/- 3.16%) (+/- 0.27%) +2.55%
For multi-threaded benchmarks, I tried kernel build and tensor flow lite. Cluster scheduling did best for the 10 thread case where 6 threads run on the P-cores, 2 threads on one Atom cluster and 2 threads on the other Atom cluster. Whereas the vanilla kernel will have 6 threads on the P-cores, 4 threads on one Atom cluster. Though the differences are small and fall within run variations.
Multi Threaded 6.3-rc5 with cluster Improvement
Benchmark scheduling in Performance
(-#threads) (run-run deviation)
-------------------------------------------------------------------------------------------
Kbuild-8 (+/- 2.90%) (+/- 1.16%) -0.76%
Kbuild-10 (+/- 3.08%) (+/- 3.09%) +0.64%
Kbuild-12 (+/- 3.28%) (+/- 3.55%) +0.91%
Tensor Lite-8 (+/- 4.84%) (+/- 4.61%) -0.23%
Tensor Lite-10 (+/- 0.87%) (+/- 1.45%) +0.47%
Tensor Lite-12 (+/- 1.37%) (+/- 1.04%) -0.12%
Thanks for reviewing these patches.
Tim Chen
Ricardo Neri (1):
sched/fair: Consider the idle state of the whole core for load balance
Tim C Chen (5):
sched/topology: Propagate SMT flags when removing degenerate domain sched/fair: Check whether active load balance is needed in busiest
group
sched/fair: Fix busiest group selection for asym groups sched/fair: Skip prefer sibling move between SMT group and non-SMT
group
sched/x86: Add cluster topology to hybrid CPU
arch/x86/kernel/smpboot.c | 3 ++ kernel/sched/fair.c | 78 ++++++++++++++++++++++++++++++++++++++- kernel/sched/topology.c | 7 +++- 3 files changed, 86 insertions(+), 2 deletions(-)
Signature · 1 line
2.32.0
PATCH 1/6 sched/topology: Propagate SMT flags when removing degenerate domain
From: Tim C Chen <[email protected]>
When a degenerate cluster domain for core with SMT CPUs is removed, the SD_SHARE_CPUCAPACITY flag in the local child sched group was not propagated to the new parent. We need this flag to properly determine whether the local sched group is SMT. Set the flag in the local child sched group of the new parent sched domain.
--- kernel/sched/topology.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
kernel/sched/topology.c +6 -1
@@ -719,8 +719,13 @@ cpu_attach_domain(struct sched_domain *sd, struct root_domain *rd, int cpu)if (sd_parent_degenerate (tmp, parent)) { tmp->parent = parent->parent; -if (parent->parent) + +if (parent->parent) { parent->parent->child = tmp; +if (tmp->flags &SD_SHARE_CPUCAPACITY ) + parent->parent->groups->flags |=SD_SHARE_CPUCAPACITY ; + } +/* * Transfer SD_PREFER_SIBLING down in case of a * degenerate parent; the spans match for this
Signature · 1 line
2.32.0
PATCH 2/6 sched/fair: Check whether active load balance is needed in busiest group
In reply to Tim ChenFrom: Tim C Chen <[email protected]>
In the busiest group, we need to consider whether active load balance to a local group is needed even when it is not overloaded. For example, when the busiest group is a SMT group that's fully busy and the destination group is a cluster group with idle CPU. Such condition is considered by asym_active_balance() in load balancing but not when looking for busiest group and load imbalance. Add this consideration in find_busiest_group() and calculate_imbalance().
--- kernel/sched/fair.c | 45 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 45 insertions(+)
kernel/sched/fair.c +45 -0
@@ -9433,6 +9433,17 @@ static inline void update_sg_lb_stats(struct lb_env *env, sgs->group_capacity; } +/* One group is SMT while the other group is not */ +static inline bool asymmetric_groups (struct sched_group *sg1, +struct sched_group *sg2) +{ +if (!sg1 || !sg2) +return false; + +return (sg1->flags &SD_SHARE_CPUCAPACITY ) != + (sg2->flags &SD_SHARE_CPUCAPACITY ); +} +/** * update_sd_pick_busiest - return 1 on busiest group * @env: The load balancing environment.
@@ -10079,6 +10090,31 @@ static inline void update_sd_lb_stats(struct lb_env *env, struct sd_lb_stats *sdupdate_idle_cpu_scan (env, sum_util); } +static inline bool asym_active_balance_busiest (struct lb_env *env,struct sd_lb_stats *sds) +{ +/* +* Don't balance to a group without spare capacity. +* +* Skip non asymmetric sched group balancing. That check +* is handled by code path handling imbalanced load between +* similar groups. +*/ +if (env->idle ==CPU_NOT_IDLE || + sds->local_stat.group_type != group_has_spare || + !asymmetric_groups (sds->local, sds->busiest)) +return false; + +/* +* For SMT source group, pull when there are two or more +* tasks over-utilizing a core. +*/ +if (sds->busiest->flags &SD_SHARE_CPUCAPACITY && + sds->busiest_stat.sum_h_nr_running >1 ) +return true; + +return false; +} +/** * calculate_imbalance - Calculate the amount of imbalance present within the * groups of a given sched_domain during load balance.
@@ -10164,6 +10200,12 @@ static inline void calculate_imbalance(struct lb_env *env, struct sd_lb_stats *sreturn ; } +if (asym_active_balance_busiest (env, sds)) { + env->migration_type = migrate_task; + env->imbalance =1 ; +return ; + } +if (busiest->group_weight ==1 || sds->prefer_sibling) {unsigned int nr_diff = busiest->sum_nr_running;/*
@@ -10371,6 +10413,9 @@ static struct sched_group *find_busiest_group(struct lb_env *env)*/ goto out_balanced; +if (asym_active_balance_busiest (env, &sds)) + goto force_balance; +if (busiest->group_weight >1 && local->idle_cpus <= (busiest->idle_cpus +1 ))/*
Signature · 1 line
2.32.0
Peter Zijlstra
maintainerRe: [PATCH 2/6] sched/fair: Check whether active load balance is needed in busiest group
Quoted 56 lines · Tim Chen
From: Tim C Chen <[email protected]>
In the busiest group, we need to consider whether active load balance to a local group is needed even when it is not overloaded. For example, when the busiest group is a SMT group that's fully busy and the destination group is a cluster group with idle CPU. Such condition is considered by
asym_active_balance()in load balancing but not when looking for busiest group and load imbalance. Add this consideration infind_busiest_group()andcalculate_imbalance().Reviewed-by Ricardo Neri <[email protected]>Signed-off-by Tim Chen <[email protected]>--- kernel/sched/fair.c | 45 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 45 insertions(+)kernel/sched/fair.c +25
-0@@ -9433,6 +9433,17 @@ static inline void update_sg_lb_stats(struct lb_env *env, sgs->group_capacity; } +/* One group is SMT while the other group is not */ +static inline bool asymmetric_groups (struct sched_group *sg1, +struct sched_group *sg2) +{ +if (!sg1 || !sg2) +return false; + +return (sg1->flags &SD_SHARE_CPUCAPACITY ) != + (sg2->flags &SD_SHARE_CPUCAPACITY ); +} +/** * update_sd_pick_busiest - return 1 on busiest group * @env: The load balancing environment. @@ -10079,6 +10090,31 @@ static inline void update_sd_lb_stats(struct lb_env *env, struct sd_lb_stats *sdupdate_idle_cpu_scan (env, sum_util); } +static inline bool asym_active_balance_busiest (struct lb_env *env,struct sd_lb_stats *sds) +{ +/* +* Don't balance to a group without spare capacity. +* +* Skip non asymmetric sched group balancing. That check +* is handled by code path handling imbalanced load between +* similar groups. +*/ +if (env->idle ==CPU_NOT_IDLE || + sds->local_stat.group_type != group_has_spare || + !asymmetric_groups (sds->local, sds->busiest)) +return false; +
kernel/sched/fair.c static inline void update_sd_lb_stats(struct lb_env *env, struct sd_lb_stats *sd · line 10116
Quoted 10 lines
kernel/sched/fair.c +10
-0+/* +* For SMT source group, pull when there are two or more +* tasks over-utilizing a core. +*/ +if (sds->busiest->flags &SD_SHARE_CPUCAPACITY && + sds->busiest_stat.sum_h_nr_running >1 ) +return true; + +return false; +}
This all seems to be mixing two 'asymmetric' things in the 'asym' namespace :/ One being the SD_ASYM_PACKING and then the above SMT/no-SMT core thing.
Quoted 18 lines
kernel/sched/fair.c +7
-0+/** * calculate_imbalance - Calculate the amount of imbalance present within the * groups of a given sched_domain during load balance. @@ -10164,6 +10200,12 @@ static inline void calculate_imbalance(struct lb_env *env, struct sd_lb_stats *sreturn ; } +if (asym_active_balance_busiest (env, sds)) { + env->migration_type = migrate_task; + env->imbalance =1 ; +return ; + } +if (busiest->group_weight ==1 || sds->prefer_sibling) {unsigned int nr_diff = busiest->sum_nr_running;/*
kernel/sched/fair.c static struct sched_group *find_busiest_group(struct lb_env *env) · line 10421
Quoted 9 lines
kernel/sched/fair.c +3
-0@@ -10371,6 +10413,9 @@ static struct sched_group *find_busiest_group(struct lb_env *env)*/ goto out_balanced; +if (asym_active_balance_busiest (env, &sds)) + goto force_balance; +if (busiest->group_weight >1 && local->idle_cpus <= (busiest->idle_cpus +1 ))/*
All the cases here have a nice (CodingStyle busting) comment, perhaps add the missing {} when hou add the comment?
Re: [PATCH 2/6] sched/fair: Check whether active load balance is needed in busiest group
Quoted 21 lines · Peter Zijlstra
Quoted 21 lines
kernel/sched/fair.c +20
-0+static inline bool asym_active_balance_busiest (struct lb_env *env,struct sd_lb_stats *sds) +{ +/* +* Don't balance to a group without spare capacity. +* +* Skip non asymmetric sched group balancing. That check +* is handled by code path handling imbalanced load between +* similar groups. +*/ +if (env->idle ==CPU_NOT_IDLE || + sds->local_stat.group_type != group_has_spare || + !asymmetric_groups (sds->local, sds->busiest)) +return false; + +/* +* For SMT source group, pull when there are two or more +* tasks over-utilizing a core. +*/ +if (sds->busiest->flags &SD_SHARE_CPUCAPACITY && + sds->busiest_stat.sum_h_nr_running >1 )
Quoted 8 lines
kernel/sched/fair.c static inline void update_sd_lb_stats(struct lb_env *env, struct sd_lb_stats *sd · line 10116
Quoted 4 lines
kernel/sched/fair.c +4
-0+return true; + +return false; +}This all seems to be mixing two 'asymmetric' things in the 'asym' namespace :/ One being the
SD_ASYM_PACKINGand then the above SMT/no-SMT core thing.
Yeah, I am kind of abusing the "asymmetric" word. However, the above code does try to set things up for the aysm_active_balance() code later. Any suggestion on better names for "asymmetric_groups()" and and "asym_active_balance_busiest()"?
Perhaps "hybrid_groups()" and "hybrid_active_balance_busiest()"?
Quoted 22 lines
Quoted 22 lines
kernel/sched/fair.c +7
-0+/** * calculate_imbalance - Calculate the amount of imbalance present within the * groups of a given sched_domain during load balance. @@ -10164,6 +10200,12 @@ static inline void calculate_imbalance(struct lb_env *env, struct sd_lb_stats *sreturn ; } +if (asym_active_balance_busiest (env, sds)) { + env->migration_type = migrate_task; + env->imbalance =1 ; +return ; + } +if (busiest->group_weight ==1 || sds->prefer_sibling) {unsigned int nr_diff = busiest->sum_nr_running;/* @@ -10371,6 +10413,9 @@ static struct sched_group *find_busiest_group(struct lb_env *env)*/ goto out_balanced;
Quoted 9 lines
kernel/sched/fair.c static struct sched_group *find_busiest_group(struct lb_env *env) · line 10421
Quoted 6 lines
kernel/sched/fair.c +3
-0+if (asym_active_balance_busiest (env, &sds)) + goto force_balance; +if (busiest->group_weight >1 && local->idle_cpus <= (busiest->idle_cpus +1 ))/* All the cases here have a nice (CodingStyle busting) comment, perhaps add the missing {} when hou add the comment?
Sure, will add a comment here.
Tim
Peter Zijlstra
maintainerRe: [PATCH 2/6] sched/fair: Check whether active load balance is needed in busiest group
Quoted 31 lines · Tim Chen
Quoted 21 lines · Peter Zijlstra
Quoted 21 lines
kernel/sched/fair.c +20
-0+static inline bool asym_active_balance_busiest (struct lb_env *env,struct sd_lb_stats *sds) +{ +/* +* Don't balance to a group without spare capacity. +* +* Skip non asymmetric sched group balancing. That check +* is handled by code path handling imbalanced load between +* similar groups. +*/ +if (env->idle ==CPU_NOT_IDLE || + sds->local_stat.group_type != group_has_spare || + !asymmetric_groups (sds->local, sds->busiest)) +return false; + +/* +* For SMT source group, pull when there are two or more +* tasks over-utilizing a core. +*/ +if (sds->busiest->flags &SD_SHARE_CPUCAPACITY && + sds->busiest_stat.sum_h_nr_running >1 )Quoted 8 lines
kernel/sched/fair.c static inline void update_sd_lb_stats(struct lb_env *env, struct sd_lb_stats *sd · line 10116
Quoted 4 lines
kernel/sched/fair.c +4
-0+return true; + +return false; +}This all seems to be mixing two 'asymmetric' things in the 'asym' namespace :/ One being the
SD_ASYM_PACKINGand then the above SMT/no-SMT core thing.
Quoted 6 lines
Yeah, I am kind of abusing the "asymmetric" word. However, the above code does try to set things up for the
aysm_active_balance()code later. Any suggestion on better names for "asymmetric_groups()" and and "asym_active_balance_busiest()"?Perhaps "
hybrid_groups()" and "hybrid_active_balance_busiest()"?
As per the other subthread; I really don't think this whole SMT vs non-SMT should be in any way shape or form be related to hybrid. Offlining siblings should really get you the same topology.
(and if that currently is not the case, that should be fixed)
(and also; we should probably add group_flags to
/debug/sched/domains/cpuN/domainM/ so we can actually see what's what, because this seems to be a bit of a blind spot).
That also suggests the hybrid naming suggestion is not a very good one.
And I'll blame it being nearly 2am for not being able to come up with a good suggestion, but I'm thinking it ought to have SMT in the name somehow.
Vincent Guittot
maintainerRe: [PATCH 2/6] sched/fair: Check whether active load balance is needed in busiest group
In reply to Tim ChenQuoted 11 lines · Tim Chen
From: Tim C Chen <[email protected]>
In the busiest group, we need to consider whether active load balance to a local group is needed even when it is not overloaded. For example, when the busiest group is a SMT group that's fully busy and the destination group is a cluster group with idle CPU. Such condition is considered by
asym_active_balance()in load balancing but not when looking for busiest group and load imbalance. Add this consideration infind_busiest_group()andcalculate_imbalance().
Quoted 2 lines
Reviewed-by Ricardo Neri <[email protected]>Signed-off-by Tim Chen <[email protected]>
Could you have a look at what we did with misfit ?
we already have a SD_ASYM_CPUCAPACITY when busiest->group_type == group_misfit_task. Misfit_task is between overloaded and ahs_spare to handle such situation. Please detect this during the update of statistic and tag the group correctly. We will not re-start to add exception everywhere.
Quoted 84 lines
--- kernel/sched/fair.c | 45 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 45 insertions(+)kernel/sched/fair.c +45
-1@@ -9433,6 +9433,17 @@ static inline void update_sg_lb_stats(struct lb_env *env, sgs->group_capacity; } +/* One group is SMT while the other group is not */ +static inline bool asymmetric_groups (struct sched_group *sg1, +struct sched_group *sg2) +{ +if (!sg1 || !sg2) +return false; + +return (sg1->flags &SD_SHARE_CPUCAPACITY ) != + (sg2->flags &SD_SHARE_CPUCAPACITY ); +} +/** * update_sd_pick_busiest - return 1 on busiest group * @env: The load balancing environment. @@ -10079,6 +10090,31 @@ static inline void update_sd_lb_stats(struct lb_env *env, struct sd_lb_stats *sdupdate_idle_cpu_scan (env, sum_util); } +static inline bool asym_active_balance_busiest (struct lb_env *env,struct sd_lb_stats *sds) +{ +/* +* Don't balance to a group without spare capacity. +* +* Skip non asymmetric sched group balancing. That check +* is handled by code path handling imbalanced load between +* similar groups. +*/ +if (env->idle ==CPU_NOT_IDLE || + sds->local_stat.group_type != group_has_spare || + !asymmetric_groups (sds->local, sds->busiest)) +return false; + +/* +* For SMT source group, pull when there are two or more +* tasks over-utilizing a core. +*/ +if (sds->busiest->flags &SD_SHARE_CPUCAPACITY && + sds->busiest_stat.sum_h_nr_running >1 ) +return true; + +return false; +} +/** * calculate_imbalance - Calculate the amount of imbalance present within the * groups of a given sched_domain during load balance. @@ -10164,6 +10200,12 @@ static inline void calculate_imbalance(struct lb_env *env, struct sd_lb_stats *sreturn ; } +if (asym_active_balance_busiest (env, sds)) { + env->migration_type = migrate_task; + env->imbalance =1 ; +return ; + } +if (busiest->group_weight ==1 || sds->prefer_sibling) {unsigned int nr_diff = busiest->sum_nr_running;/* @@ -10371,6 +10413,9 @@ static struct sched_group *find_busiest_group(struct lb_env *env)*/ goto out_balanced; +if (asym_active_balance_busiest (env, &sds)) + goto force_balance; +if (busiest->group_weight >1 && local->idle_cpus <= (busiest->idle_cpus +1 ))/* -- 2.32.0
PATCH 3/6 sched/fair: Fix busiest group selection for asym groups
In reply to Tim ChenFrom: Tim C Chen <[email protected]>
For two groups that have spare capacity and are partially busy, the busier group should be the group with pure CPUs rather than the group with SMT CPUs. We do not want to make the SMT group the busiest one to pull task off the SMT and makes the whole core empty.
Otherwise suppose in the search for busiest group,
we first encounter an SMT group with 1 task and set it as the busiest. The local group is an atom cluster with 1 task and we then encounter an atom cluster group with 3 tasks, we will not pick this atom cluster group over the SMT group, even though we should. As a result, we do not load balance the busier Atom cluster (with 3 tasks) towards the local Atom cluster (with 1 task). And it doesn't make sense to pick the 1 task SMT group as the busier group as we also should not pull task off the SMT towards the 1 task atom cluster and make the SMT core completely empty.
Fix this case.
--- kernel/sched/fair.c | 12 ++++++++++++ 1 file changed, 12 insertions(+)
kernel/sched/fair.c +12 -0
@@ -9548,6 +9548,18 @@ static bool update_sd_pick_busiest(struct lb_env *env,break ;case group_has_spare: +/* +* Do not pick sg with SMT CPUs over sg with pure CPUs, +* as we do not want to pull task off half empty SMT core +* and make the core idle. +*/ +if (asymmetric_groups (sds->busiest, sg)) { +if (sds->busiest->flags &SD_SHARE_CPUCAPACITY ) +return true; +else +return false; + } +/* * Select not overloaded group with lowest number of idle cpus * and highest number of running tasks. We could also compare
Signature · 1 line
2.32.0
Peter Zijlstra
maintainerRe: [PATCH 3/6] sched/fair: Fix busiest group selection for asym groups
Quoted 30 lines · Tim Chen
From: Tim C Chen <[email protected]>
For two groups that have spare capacity and are partially busy, the busier group should be the group with pure CPUs rather than the group with SMT CPUs. We do not want to make the SMT group the busiest one to pull task off the SMT and makes the whole core empty.
Otherwise suppose in the search for busiest group,
we first encounter an SMT group with 1 task and set it as the busiest. The local group is an atom cluster with 1 task and we then encounter an atom cluster group with 3 tasks, we will not pick this atom cluster group over the SMT group, even though we should. As a result, we do not load balance the busier Atom cluster (with 3 tasks) towards the local Atom cluster (with 1 task). And it doesn't make sense to pick the 1 task SMT group as the busier group as we also should not pull task off the SMT towards the 1 task atom cluster and make the SMT core completely empty.Fix this case.
Reviewed-by Ricardo Neri <[email protected]>Signed-off-by Tim Chen <[email protected]>--- kernel/sched/fair.c | 12 ++++++++++++ 1 file changed, 12 insertions(+)kernel/sched/fair.c +0
-0
kernel/sched/fair.c static bool update_sd_pick_busiest(struct lb_env *env, · line 9555
Quoted 8 lines
kernel/sched/fair.c +5
-0@@ -9548,6 +9548,18 @@ static bool update_sd_pick_busiest(struct lb_env *env,break ;case group_has_spare: +/* +* Do not pick sg with SMT CPUs over sg with pure CPUs, +* as we do not want to pull task off half empty SMT core +* and make the core idle. +*/
Comment says what the code does; not why.
kernel/sched/fair.c static bool update_sd_pick_busiest(struct lb_env *env, · line 9560
Quoted 5 lines
kernel/sched/fair.c +5
-0+if (asymmetric_groups (sds->busiest, sg)) { +if (sds->busiest->flags &SD_SHARE_CPUCAPACITY ) +return true; +else +return false;
return (sds->busiest->flags & SD_SHARE_CPUCAPACITY)
kernel/sched/fair.c static bool update_sd_pick_busiest(struct lb_env *env, · line 9561
Quoted 1 line
kernel/sched/fair.c +1
-0+ }
Also, should this not be part of the previous patch?
Re: [PATCH 3/6] sched/fair: Fix busiest group selection for asym groups
Quoted 10 lines · Peter Zijlstra
kernel/sched/fair.c static bool update_sd_pick_busiest(struct lb_env *env, · line 9555
Quoted 8 lines
kernel/sched/fair.c +5
-0case group_has_spare: +/* +* Do not pick sg with SMT CPUs over sg with pure CPUs, +* as we do not want to pull task off half empty SMT core +* and make the core idle. +*/ Comment says what the code does; not why.
Good point, will make the comment better.
Quoted 11 lines
kernel/sched/fair.c static bool update_sd_pick_busiest(struct lb_env *env, · line 9560
Quoted 6 lines
kernel/sched/fair.c +5
-0+if (asymmetric_groups (sds->busiest, sg)) { +if (sds->busiest->flags &SD_SHARE_CPUCAPACITY ) +return true; +else +return false;return (sds->busiest->flags & SD_SHARE_CPUCAPACITY)kernel/sched/fair.c static bool update_sd_pick_busiest(struct lb_env *env, · line 9561
Quoted 1 line
kernel/sched/fair.c +1
-0+ }Also, should this not be part of the previous patch?
Sure, I can merge it with the previous patch.
Tim
PATCH 4/6 sched/fair: Skip prefer sibling move between SMT group and non-SMT group
In reply to Tim ChenFrom: Tim C Chen <[email protected]>
Do not try to move tasks between non SMT sched group and SMT sched group for "prefer sibling" load balance.
Let asym_active_balance_busiest() handle that case properly. Otherwise we could get task bouncing back and forth between the SMT sched group and non SMT sched group.
--- kernel/sched/fair.c | 4 ++++ 1 file changed, 4 insertions(+)
kernel/sched/fair.c +4 -0
@@ -10411,8 +10411,12 @@ static struct sched_group *find_busiest_group(struct lb_env *env)/* * Try to move all excess tasks to a sibling domain of the busiest * group's child domain. +* +* Do not try to move between non smt sched group and smt sched +* group. Let asym active balance properly handle that case. */ if (sds.prefer_sibling && local->group_type == group_has_spare && + !asymmetric_groups (sds.busiest, sds.local) && busiest->sum_nr_running > local->sum_nr_running +1 ) goto force_balance;
Signature · 1 line
2.32.0
Peter Zijlstra
maintainerRe: [PATCH 4/6] sched/fair: Skip prefer sibling move between SMT group and non-SMT group
Quoted 20 lines · Tim Chen
From: Tim C Chen <[email protected]>
Do not try to move tasks between non SMT sched group and SMT sched group for "prefer sibling" load balance.
Letasym_active_balance_busiest()handle that case properly. Otherwise we could get task bouncing back and forth between the SMT sched group and non SMT sched group.Reviewed-by Ricardo Neri <[email protected]>Signed-off-by Tim Chen <[email protected]>--- kernel/sched/fair.c | 4 ++++ 1 file changed, 4 insertions(+)kernel/sched/fair.c +0
-0@@ -10411,8 +10411,12 @@ static struct sched_group *find_busiest_group(struct lb_env *env)/*
kernel/sched/fair.c static struct sched_group *find_busiest_group(struct lb_env *env) · line 10421
Quoted 10 lines
kernel/sched/fair.c +4
-0* Try to move all excess tasks to a sibling domain of the busiest * group's child domain. +* +* Do not try to move between non smt sched group and smt sched +* group. Let asym active balance properly handle that case. */ if (sds.prefer_sibling && local->group_type == group_has_spare && + !asymmetric_groups (sds.busiest, sds.local) && busiest->sum_nr_running > local->sum_nr_running +1 ) goto force_balance;
This seems to have the hidden assumption that a !SMT core is somehow 'less' that an SMT code. Should this not also look at sched_asym_prefer() to establush this is so?
I mean, imagine I have a regular system and just offline one smt sibling for giggles.
Re: [PATCH 4/6] sched/fair: Skip prefer sibling move between SMT group and non-SMT group
Quoted 36 lines · Peter Zijlstra
Quoted 20 lines · Tim Chen
From: Tim C Chen <[email protected]>
Do not try to move tasks between non SMT sched group and SMT sched group for "prefer sibling" load balance.
Letasym_active_balance_busiest()handle that case properly. Otherwise we could get task bouncing back and forth between the SMT sched group and non SMT sched group.Reviewed-by Ricardo Neri <[email protected]>Signed-off-by Tim Chen <[email protected]>--- kernel/sched/fair.c | 4 ++++ 1 file changed, 4 insertions(+)kernel/sched/fair.c +0
-0@@ -10411,8 +10411,12 @@ static struct sched_group *find_busiest_group(struct lb_env *env)/* kernel/sched/fair.c static struct sched_group *find_busiest_group(struct lb_env *env) · line 10421
Quoted 10 lines
kernel/sched/fair.c +4
-0* Try to move all excess tasks to a sibling domain of the busiest * group's child domain. +* +* Do not try to move between non smt sched group and smt sched +* group. Let asym active balance properly handle that case. */ if (sds.prefer_sibling && local->group_type == group_has_spare && + !asymmetric_groups (sds.busiest, sds.local) && busiest->sum_nr_running > local->sum_nr_running +1 ) goto force_balance;This seems to have the hidden assumption that a !SMT core is somehow 'less' that an SMT code. Should this not also look at
sched_asym_prefer()to establush this is so?
Quoted 2 lines
I mean, imagine I have a regular system and just offline one smt sibling for giggles.
I don't quite follow your point as asymmetric_groups() returns false even one smt sibling is offlined.
Even say sds.busiest has 1 SMT and sds.local has 2 SMT, both sched groups still have SD_SHARE_CPUCAPACITY flag turned on. So asymmetric_groups() return false and the load balancing logic is not changed for regular non-hybrid system.
I may be missing something.
Tim
Peter Zijlstra
maintainerRe: [PATCH 4/6] sched/fair: Skip prefer sibling move between SMT group and non-SMT group
Quoted 40 lines · Tim Chen
Quoted 36 lines · Peter Zijlstra
Quoted 20 lines · Tim Chen
From: Tim C Chen <[email protected]>
Do not try to move tasks between non SMT sched group and SMT sched group for "prefer sibling" load balance.
Letasym_active_balance_busiest()handle that case properly. Otherwise we could get task bouncing back and forth between the SMT sched group and non SMT sched group.Reviewed-by Ricardo Neri <[email protected]>Signed-off-by Tim Chen <[email protected]>--- kernel/sched/fair.c | 4 ++++ 1 file changed, 4 insertions(+)kernel/sched/fair.c +0
-0@@ -10411,8 +10411,12 @@ static struct sched_group *find_busiest_group(struct lb_env *env)/* kernel/sched/fair.c static struct sched_group *find_busiest_group(struct lb_env *env) · line 10421
Quoted 10 lines
kernel/sched/fair.c +4
-0* Try to move all excess tasks to a sibling domain of the busiest * group's child domain. +* +* Do not try to move between non smt sched group and smt sched +* group. Let asym active balance properly handle that case. */ if (sds.prefer_sibling && local->group_type == group_has_spare && + !asymmetric_groups (sds.busiest, sds.local) && busiest->sum_nr_running > local->sum_nr_running +1 ) goto force_balance;This seems to have the hidden assumption that a !SMT core is somehow 'less' that an SMT code. Should this not also look at
sched_asym_prefer()to establush this is so?Quoted 2 lines
I mean, imagine I have a regular system and just offline one smt sibling for giggles.
Quoted 8 lines
I don't quite follow your point as
asymmetric_groups()returns false even one smt sibling is offlined.Even say sds.busiest has 1 SMT and sds.local has 2 SMT, both sched groups still have
SD_SHARE_CPUCAPACITYflag turned on. Soasymmetric_groups()return false and the load balancing logic is not changed for regular non-hybrid system.I may be missing something.
What's the difference between the two cases? That is, if the remaining sibling will have SD_SHARE_CPUCAPACIY from the degenerate SMT domain that's been reaped, then why doesn't the same thing apply to the atoms in the hybrid muck?
Those two cases *should* be identical, both cases you have cores with and cores without SMT.
Peter Zijlstra
maintainerRe: [PATCH 4/6] sched/fair: Skip prefer sibling move between SMT group and non-SMT group
Quoted 50 lines · Peter Zijlstra
Quoted 40 lines · Tim Chen
Quoted 36 lines · Peter Zijlstra
Quoted 20 lines · Tim Chen
From: Tim C Chen <[email protected]>
Do not try to move tasks between non SMT sched group and SMT sched group for "prefer sibling" load balance.
Letasym_active_balance_busiest()handle that case properly. Otherwise we could get task bouncing back and forth between the SMT sched group and non SMT sched group.Reviewed-by Ricardo Neri <[email protected]>Signed-off-by Tim Chen <[email protected]>--- kernel/sched/fair.c | 4 ++++ 1 file changed, 4 insertions(+)kernel/sched/fair.c +0
-0@@ -10411,8 +10411,12 @@ static struct sched_group *find_busiest_group(struct lb_env *env)/* kernel/sched/fair.c static struct sched_group *find_busiest_group(struct lb_env *env) · line 10421
Quoted 10 lines
kernel/sched/fair.c +4
-0* Try to move all excess tasks to a sibling domain of the busiest * group's child domain. +* +* Do not try to move between non smt sched group and smt sched +* group. Let asym active balance properly handle that case. */ if (sds.prefer_sibling && local->group_type == group_has_spare && + !asymmetric_groups (sds.busiest, sds.local) && busiest->sum_nr_running > local->sum_nr_running +1 ) goto force_balance;This seems to have the hidden assumption that a !SMT core is somehow 'less' that an SMT code. Should this not also look at
sched_asym_prefer()to establush this is so?Quoted 2 lines
I mean, imagine I have a regular system and just offline one smt sibling for giggles.
Quoted 8 lines
I don't quite follow your point as
asymmetric_groups()returns false even one smt sibling is offlined.Even say sds.busiest has 1 SMT and sds.local has 2 SMT, both sched groups still have
SD_SHARE_CPUCAPACITYflag turned on. Soasymmetric_groups()return false and the load balancing logic is not changed for regular non-hybrid system.I may be missing something.
Quoted 7 lines
What's the difference between the two cases? That is, if the remaining sibling will have
SD_SHARE_CPUCAPACIYfrom the degenerate SMT domain that's been reaped, then why doesn't the same thing apply to the atoms in the hybrid muck?Those two cases *should* be identical, both cases you have cores with and cores without SMT.
On my alderlake:
[ 202.222019] CPU0 attaching sched-domain(s):
[ 202.222509] domain-0: span=0-1 level=SMT
[ 202.222707] groups: 0:{ span=0 }, 1:{ span=1 }
[ 202.222945] domain-1: span=0-23 level=MC
[ 202.223148] groups: 0:{ span=0-1 cap=2048 }, 2:{ span=2-3 cap=2048 }, 4:{ span=4-5 cap=2048 }, 6:{ span=6-7 cap=2048 }, 8:{ span=8-9 cap=2048 }, 10:{ span=10-11 cap=2048 },12:{ span=12-13 cap=2048 }, 14:{ span=14-15 cap=2048 }, 16:{ span=16 }, 17:{ span=17 }, 18:{ span=18 }, 19:{ span=19 }, 20:{ span=20 }, 21:{ span=21 }, 22:{ span=22 }, 23:{ span=23 }
...
[ 202.249979] CPU23 attaching sched-domain(s):
[ 202.250127] domain-0: span=0-23 level=MC
[ 202.250198] groups: 23:{ span=23 }, 0:{ span=0-1 cap=2048 }, 2:{ span=2-3 cap=2048 }, 4:{ span=4-5 cap=2048 }, 6:{ span=6-7 cap=2048 }, 8:{ span=8-9 cap=2048 }, 10:{ span=10-11 cap=2048 }, 12:{ span=12-13 cap=2048 }, 14:{ span=14-15 cap=2048 }, 16:{ span=16 }, 17:{ span=17 }, 18:{ span=18 }, 19:{ span=19 }, 20:{ span=20 }, 21:{ span=21 }, 22:{ span=22 }$ echo 0 > /sys/devices/system/cpu/cpu1/online
[ 251.213848] CPU0 attaching sched-domain(s):
[ 251.214376] domain-0: span=0,2-23 level=MC
[ 251.214580] groups: 0:{ span=0 }, 2:{ span=2-3 cap=2048 }, 4:{ span=4-5 cap=2048 }, 6:{ span=6-7 cap=2048 }, 8:{ span=8-9 cap=2048 }, 10:{ span=10-11 cap=2048 }, 12:{ span=12-13 cap=2048 }, 14:{ span=14-15 cap=2048 }, 16:{ span=16 }, 17:{ span=17 }, 18:{ span=18 }, 19:{ span=19 }, 20:{ span=20 }, 21:{ span=21 }, 22:{ span=22 }, 23:{ span=23 }
...
[ 251.239511] CPU23 attaching sched-domain(s):
[ 251.239656] domain-0: span=0,2-23 level=MC
[ 251.239727] groups: 23:{ span=23 }, 0:{ span=0 }, 2:{ span=2-3 cap=2048 }, 4:{ span=4-5 cap=2048 }, 6:{ span=6-7 cap=2048 }, 8:{ span=8-9 cap=2048 }, 10:{ span=10-11 cap=2048 }, 12:{ span=12-13 cap=2048 }, 14:{ span=14-15 cap=2048 }, 16:{ span=16 }, 17:{ span=17 }, 18:{ span=18 }, 19:{ span=19 }, 20:{ span=20 }, 21:{ span=21 }, 22:{ span=22 }$ cat /debug/sched/domains/cpu0/domain0/groups_flags
$ cat /debug/sched/domains/cpu23/domain0/groups_flags
IOW, neither the big core with SMT with one sibling offline, nor the little core with no SMT on at all have the relevant flags set on their domain0 groups.
---
kernel/sched/debug.c +1 -0
@@ -427,6 +427,7 @@ static void register_sd(struct sched_domain *sd, struct dentry *parent)#undef SDMdebugfs_create_file ("flags" ,0444 , parent, &sd->flags, &sd_flags_fops); +debugfs_create_file ("groups_flags" ,0444 , parent, &sd->groups->flags, &sd_flags_fops); }void update_sched_domain_debugfs (void )
Vincent Guittot
maintainerRe: [PATCH 4/6] sched/fair: Skip prefer sibling move between SMT group and non-SMT group
In reply to Tim ChenQuoted 20 lines · Tim Chen
From: Tim C Chen <[email protected]>
Do not try to move tasks between non SMT sched group and SMT sched group for "prefer sibling" load balance.
Letasym_active_balance_busiest()handle that case properly. Otherwise we could get task bouncing back and forth between the SMT sched group and non SMT sched group.Reviewed-by Ricardo Neri <[email protected]>Signed-off-by Tim Chen <[email protected]>--- kernel/sched/fair.c | 4 ++++ 1 file changed, 4 insertions(+)kernel/sched/fair.c +0
-0
kernel/sched/fair.c static struct sched_group *find_busiest_group(struct lb_env *env) · line 10419
Quoted 9 lines
kernel/sched/fair.c +4
-0@@ -10411,8 +10411,12 @@ static struct sched_group *find_busiest_group(struct lb_env *env)/* * Try to move all excess tasks to a sibling domain of the busiest * group's child domain. +* +* Do not try to move between non smt sched group and smt sched +* group. Let asym active balance properly handle that case. */ if (sds.prefer_sibling && local->group_type == group_has_spare && + !asymmetric_groups(sds.busiest, sds.local) &&
Can't you delete SD_PREFER_SIBLING flags when building topology like SD_ASYM_CPUCAPACITY does ?
Generally speaking SD_ASYM_CPUCAPACITY and SD_ASYM_PACKING are doing quite similar thing, it would be good to get one common solution instead 2 parallel paths
Quoted 6 lines
kernel/sched/fair.c +0
-1busiest->sum_nr_running > local->sum_nr_running + 1) goto force_balance; --2.32.0
Re: [PATCH 4/6] sched/fair: Skip prefer sibling move between SMT group and non-SMT group
Quoted 12 lines · Vincent Guittot
Quoted 11 lines · Tim Chen
kernel/sched/fair.c +1
-0@@ -10411,8 +10411,12 @@ static struct sched_group *find_busiest_group(struct lb_env *env)/* * Try to move all excess tasks to a sibling domain of the busiest * group's child domain. +*
Quoted 8 lines
kernel/sched/fair.c static struct sched_group *find_busiest_group(struct lb_env *env) · line 10419
Quoted 5 lines
kernel/sched/fair.c +3
-0+* Do not try to move between non smt sched group and smt sched +* group. Let asym active balance properly handle that case. */ if (sds.prefer_sibling && local->group_type == group_has_spare && + !asymmetric_groups(sds.busiest, sds.local) &&Can't you delete
SD_PREFER_SIBLINGflags when building topology likeSD_ASYM_CPUCAPACITYdoes ?
The sched domain actually can have a mixture of sched groups with Atom modules and sched groups with SMT cores. When comparing sched group of Atom core cluster and Atom core cluster, or SMT core with SMT core, I think we do want the prefer sibling logic. It is only when we are comparing SMT core and Atom core cluster we
want to skip this. Ricardo, please correct me if I am wrong.
Quoted 4 lines
Generally speaking
SD_ASYM_CPUCAPACITYandSD_ASYM_PACKINGare doing quite similar thing, it would be good to get one common solution instead 2 parallel paths
Okay. I'll see what I can do to merge the handling.
Tim
Quoted 7 lines
Quoted 7 lines
kernel/sched/fair.c +0
-1busiest->sum_nr_running > local->sum_nr_running + 1) goto force_balance; --2.32.0
PATCH 5/6 sched/fair: Consider the idle state of the whole core for load balance
In reply to Tim ChenFrom: Ricardo Neri <[email protected]>
should_we_balance() traverses the group_balance_mask (AND'ed with lb_env::
cpus) starting from lower numbered CPUs looking for the first idle CPU.
In hybrid x86 systems, the siblings of SMT cores get CPU numbers, before non-SMT cores:
[0, 1] [2, 3] [4, 5] 5 6 7 8
b i b i b i b i i iIn the figure above, CPUs in brackets are siblings of an SMT core. The rest are non-SMT cores. 'b' indicates a busy CPU, 'i' indicates an idle CPU.
We should let a CPU on a fully idle core get the first chance to idle load balance as it has more CPU capacity than a CPU on an idle SMT CPU with busy sibling. So for the figure above, if we are running should_we_balance() to CPU 1, we should return false to let CPU 6 on idle core to have a chance first to idle load balance.
A partially busy (i.e., of type group_has_spare) local group with SMT cores will often have only one SMT sibling busy. If the destination CPU is a non-SMT core, partially busy, lower-numbered, SMT cores should not be considered when finding the first idle CPU.
However, in should_we_balance(), when we encounter idle SMT first in partially busy core, we prematurely break the search for the first idle CPU.
Higher-numbered, non-SMT cores is not given the chance to have idle balance done on their behalf. Those CPUs will only be considered for idle balancing by chance via CPU_NEWLY_IDLE.
Instead, consider the idle state of the whole SMT core.
--- kernel/sched/fair.c | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-)
kernel/sched/fair.c +16 -1
@@ -10683,7 +10683,7 @@ static int active_load_balance_cpu_stop(void *data);static int should_we_balance (struct lb_env *env) {struct sched_group *sg = env->sd->groups; -int cpu; +int cpu, idle_smt =-1 ;/* * Ensure the balancing environment is consistent; can happen
@@ -10709,11 +10709,26 @@ static int should_we_balance(struct lb_env *env)for_each_cpu_and (cpu,group_balance_mask (sg), env->cpus) {if (!idle_cpu (cpu))continue ; +else { +/* +* Don't balance to idle SMT in busy core right away when +* balancing cores, but remember the first idle SMT CPU for +* later consideration. Find CPU on an idle core first. +*/ +if (!(env->sd->flags &SD_SHARE_CPUCAPACITY ) && !is_core_idle (cpu)) { +if (idle_smt ==-1 ) + idle_smt = cpu; +continue ; + } + }/* Are we the first idle CPU? */ return cpu == env->dst_cpu; } +if (idle_smt == env->dst_cpu) +return true; +/* Are we the first CPU of this group ? */ return group_balance_cpu (sg) == env->dst_cpu; }
Signature · 1 line
2.32.0
Peter Zijlstra
maintainerRe: [PATCH 5/6] sched/fair: Consider the idle state of the whole core for load balance
Quoted 6 lines · Tim Chen
kernel/sched/fair.c +2
-0@@ -10709,11 +10709,26 @@ static int should_we_balance(struct lb_env *env)for_each_cpu_and (cpu,group_balance_mask (sg), env->cpus) {if (!idle_cpu (cpu))continue ; +else { +/*
kernel/sched/fair.c static int should_we_balance(struct lb_env *env) · line 10723
Quoted 10 lines
kernel/sched/fair.c +10
-0+* Don't balance to idle SMT in busy core right away when +* balancing cores, but remember the first idle SMT CPU for +* later consideration. Find CPU on an idle core first. +*/ +if (!(env->sd->flags &SD_SHARE_CPUCAPACITY ) && !is_core_idle (cpu)) { +if (idle_smt ==-1 ) + idle_smt = cpu; +continue ; + } + }
Not only does that bust CodingStyle, it's also entirely daft. What exactly is the purpose of that else statement?
Quoted 4 lines
kernel/sched/fair.c +0
-0/* Are we the first idle CPU? */ return cpu == env->dst_cpu; }
Re: [PATCH 5/6] sched/fair: Consider the idle state of the whole core for load balance
Quoted 13 lines · Peter Zijlstra
Quoted 11 lines · Tim Chen
kernel/sched/fair.c +7
-0@@ -10709,11 +10709,26 @@ static int should_we_balance(struct lb_env *env)for_each_cpu_and (cpu,group_balance_mask (sg), env->cpus) {if (!idle_cpu (cpu))continue ; +else { +/* +* Don't balance to idle SMT in busy core right away when +* balancing cores, but remember the first idle SMT CPU for +* later consideration. Find CPU on an idle core first. +*/ +if (!(env->sd->flags &SD_SHARE_CPUCAPACITY ) && !is_core_idle (cpu)) {
Quoted 10 lines
kernel/sched/fair.c static int should_we_balance(struct lb_env *env) · line 10723
Quoted 5 lines
kernel/sched/fair.c +5
-0+if (idle_smt ==-1 ) + idle_smt = cpu; +continue ; + } + }Not only does that bust CodingStyle, it's also entirely daft. What exactly is the purpose of that else statement?
Yeah, that's a dumb "else" statement. Will remove that.
Tim
PATCH 6/6 sched/x86: Add cluster topology to hybrid CPU
In reply to Tim ChenFrom: Tim C Chen <[email protected]>
Cluster topology was not enabled on hybrid x86 CPU as load balance was not properly working for cluster domain. That has been fixed and cluster domain can be enabled for hybrid CPU.
--- arch/x86/kernel/smpboot.c | 3 +++ 1 file changed, 3 insertions(+)
arch/x86/kernel/smpboot.c +3 -0
@@ -575,6 +575,9 @@ static struct sched_domain_topology_level x86_hybrid_topology[] = { #ifdef CONFIG_SCHED_SMT { cpu_smt_mask, x86_smt_flags, SD_INIT_NAME(SMT) }, #endif +#ifdef CONFIG_SCHED_CLUSTER + { cpu_clustergroup_mask, x86_cluster_flags, SD_INIT_NAME(CLS) }, +#endif #ifdef CONFIG_SCHED_MC { cpu_coregroup_mask, x86_core_flags, SD_INIT_NAME(MC) }, #endif
Signature · 1 line
2.32.0
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: [PATCH 5/6] sched/fair: Consider the idle state of the whole core for load balance
Quoted 12 lines · Hillf Danton
On 4 May 2023 09:09:55 -0700 Tim Chen <[email protected]>
Quoted 11 lines
From: Ricardo Neri <[email protected]>
should_we_balance()traverses thegroup_balance_mask(AND'ed withlb_env::
cpus) starting from lower numbered CPUs looking for the first idle CPU.In hybrid x86 systems, the siblings of SMT cores get CPU numbers, before non-SMT cores:
[0, 1] [2, 3] [4, 5] 5 6 7 8 b i b i b i b i i i
Quoted 7 lines
Quoted 3 lines
In the figure above, CPUs in brackets are siblings of an SMT core. The rest are non-SMT cores. 'b' indicates a busy CPU, 'i' indicates an idle CPU.
Better if l2-cache affinity is added in the diagram above. And better again if the diagram is available upon introducing hybrid x86 in the cover letter.
Good suggestion. Will update the diagram and add it to cover letter.
Tim