x86/sched: Avoid unnecessary migrations within SMT domains
v2 cover x86/sched: Avoid unnecessary migrations within SMT domains
Hi,
This v2 of this patchset. V1 can be found here [1]. In this version I took the suggestion of Peter to teach arch_asym_cpu_priority() the CPU state. Also, I reworded the cover letter to explain better the intent.
---
asym_packing load balancing is used to balance load among physical cores with SMT (e.g., Intel processors that support Intel Turbo Boost Max 3.0 and hybrid processors) and among SMT siblings of a physical cores (e.g., Power7).
The current implementation is sufficient for the latter case as it favors higher-priority SMT siblings. In the former case, however, we must consider the fact that the throughput of a CPU degrades if one or more of its SMT siblings are busy. Hence, a lower-priority CPU that is fully idle is more desirable than a high-priority CPU with busy SMT siblings.
To fit the current implementation of asym_packing, x86 artificially assigns a lower priority to the higher-numbered SMT siblings. In reality, there is no difference between any of the SMT siblings of a core.
Do not use different priorities for each SMT sibling. Instead, tweak the asym_packing load balancing logic to consider the idle state of the SMT siblings of a CPU.
Removing these artificial priorities avoids superfluous migrations and let lower-priority cores inspect all SMT siblings for the busiest queue. The latter is also necessary to support IPC classes of tasks [2], as the destination CPU will need to inspect the tasks running on CPUs of equal priority.
This patchset should not break Power7 SMT8. Functionality does not change for architectures that do not implement the new check_smt parameter of sched_prefer_asym().
These patches apply cleanly on today's tip tree.
Changes since v1:
* Tweaked arch_asym_cpu_priority() and sched_asym_prefer() to handle the idle state of the SMT siblings of a CPU. (PeterZ)
* Expose functionality of the scheduler that determines the idle state of the SMT siblings of a CPU.
* Addressed concerns from Peter about SMT2 assumptions and breaking Power7.
* Removed the SD_ASYM_PACKING flag from the "SMT" domain in x86.
* Reworked x86's arch_asym_cpu_priority() to consider the idle state of the SMT siblings of a CPU.
[1]. https://lore.kernel.org/lkml/[email protected]/
[2]. https://lore.kernel.org/lkml/[email protected]/
Ricardo Neri (7):
sched/fair: Generalize asym_packing logic for SMT local sched group
sched: Prepare sched_asym_prefer() to handle idle state of SMT
siblings
sched: Teach arch_asym_cpu_priority() the idle state of SMT siblings sched/fair: Introduce sched_smt_siblings_idle() x86/sched: Remove SD_ASYM_PACKING from the "SMT" domain x86/sched/itmt: Give all SMT siblings of a core the same priority x86/sched/itmt: Consider the idle state of SMT siblings
arch/x86/kernel/itmt.c | 30 ++++-------- arch/x86/kernel/smpboot.c | 2 +- include/linux/sched.h | 2 + include/linux/sched/topology.h | 2 +- kernel/sched/fair.c | 90 +++++++++++++++++----------------- kernel/sched/sched.h | 11 +++-- kernel/sched/topology.c | 6 ++- 7 files changed, 72 insertions(+), 71 deletions(-)
Signature · 1 line
2.25.1
PATCH v2 1/7 sched/fair: Generalize asym_packing logic for SMT local sched group
When balancing load between two physical cores, an idle destination CPU can help another core only if all of its SMT siblings are also idle. Otherwise, there is not increase in throughput. It does not matter whether the other core is composed of SMT siblings.
Simply check if there are any tasks running on the local group and the other core has exactly one busy CPU before proceeding. Let find_busiest_group() handle the case of more than one busy CPU. This is true for SMT2, SMT4, SMT8, etc.
---
Changes since v1:
* Reworded commit message and inline comments for clarity.
* Stated that this changeset does not impact STM4 or SMT8.
--- kernel/sched/fair.c | 29 +++++++++-------------------- 1 file changed, 9 insertions(+), 20 deletions(-)
kernel/sched/fair.c +9 -20
@@ -8900,12 +8900,10 @@ static bool asym_smt_can_pull_tasks(int dst_cpu, struct sd_lb_stats *sds, struct sched_group *sg) {#ifdef CONFIG_SCHED_SMT -bool local_is_smt, sg_is_smt; +bool local_is_smt;int sg_busy_cpus; local_is_smt = sds->local->flags &SD_SHARE_CPUCAPACITY ; - sg_is_smt = sg->flags &SD_SHARE_CPUCAPACITY ; - sg_busy_cpus = sgs->group_weight - sgs->idle_cpus;if (!local_is_smt) {
@@ -8926,25 +8924,16 @@ static bool asym_smt_can_pull_tasks(int dst_cpu, struct sd_lb_stats *sds,return sched_asym_prefer (dst_cpu, sg->asym_prefer_cpu); } -/* @dst_cpu has SMT siblings. */ - -if (sg_is_smt) { -int local_busy_cpus = sds->local->group_weight - - sds->local_stat.idle_cpus; -int busy_cpus_delta = sg_busy_cpus - local_busy_cpus; - -if (busy_cpus_delta ==1 ) -return sched_asym_prefer (dst_cpu, sg->asym_prefer_cpu); - -return false; - } -/* -* @sg does not have SMT siblings. Ensure that @sds::local does not end -* up with more than one busy SMT sibling and only pull tasks if there -* are not busy CPUs (i.e., no CPU has running tasks). +* @dst_cpu has SMT siblings. Do asym_packing load balancing only if +* all its siblings are idle (moving tasks between physical cores in +* which some SMT siblings are busy results in the same throughput). +* +* If the difference in the number of busy CPUs is two or more, let +* find_busiest_group() take care of it. We only care if @sg has +* exactly one busy CPU. This covers SMT and non-SMT sched groups. */ -if (!sds->local_stat.sum_nr_running) +if (sg_busy_cpus ==1 && !sds->local_stat.sum_nr_running)return sched_asym_prefer (dst_cpu, sg->asym_prefer_cpu);return false;
Signature · 1 line
2.25.1
Dietmar Eggemann
reviewerRe: [PATCH v2 1/7] sched/fair: Generalize asym_packing logic for SMT local sched group
Quoted 9 lines · Ricardo Neri
When balancing load between two physical cores, an idle destination CPU can help another core only if all of its SMT siblings are also idle. Otherwise, there is not increase in throughput. It does not matter whether the other core is composed of SMT siblings.
Simply check if there are any tasks running on the local group and the other core has exactly one busy CPU before proceeding. Let
find_busiest_group()handle the case of more than one busy CPU. This is true for SMT2, SMT4, SMT8, etc.
[...]
kernel/sched/fair.c static bool asym_smt_can_pull_tasks(int dst_cpu, struct sd_lb_stats *sds, · line 8900
Quoted 13 lines
Changes since v1:
* Reworded commit message and inline comments for clarity.
* Stated that this changeset does not impact STM4 or SMT8.--- kernel/sched/fair.c | 29 +++++++++-------------------- 1 file changed, 9 insertions(+), 20 deletions(-)kernel/sched/fair.c +0
-0@@ -8900,12 +8900,10 @@ static bool asym_smt_can_pull_tasks(int dst_cpu, struct sd_lb_stats *sds, struct sched_group *sg)
I'm not sure why you change asym_smt_can_pull_tasks() together with removing SD_ASYM_PACKING from SMT level (patch 5/7)?
update_sg_lb_stats()
... && env->sd->flags & SD_ASYM_PACKING && .. && sched_asym()
^^^^^^^^^^^^
sched_asym() if ((sds->local->flags & SD_SHARE_CPUCAPACITY) ||
(group->flags & SD_SHARE_CPUCAPACITY))
return asym_smt_can_pull_tasks()
^^^^^^^^^^^^^^^^^^^^^^^^^So x86 won't have a sched domain with SD_SHARE_CPUCAPACITY and SD_ASYM_PACKING anymore. So sched_asym() would call sched_asym_prefer() directly on MC. What do I miss here?
[...]
Re: [PATCH v2 1/7] sched/fair: Generalize asym_packing logic for SMT local sched group
Quoted 6 lines · Dietmar Eggemann
Quoted 5 lines · Ricardo Neri
When balancing load between two physical cores, an idle destination CPU can help another core only if all of its SMT siblings are also idle. Otherwise, there is not increase in throughput. It does not matter whether the other core is composed of SMT siblings.
Quoted 6 lines
Quoted 4 lines
Simply check if there are any tasks running on the local group and the other core has exactly one busy CPU before proceeding. Let
find_busiest_group()handle the case of more than one busy CPU. This is true for SMT2, SMT4, SMT8, etc.[...]
Thank you very much for your feedback, Dietmar!
Quoted 8 lines
Quoted 8 lines
Changes since v1:
* Reworded commit message and inline comments for clarity.
* Stated that this changeset does not impact STM4 or SMT8.--- kernel/sched/fair.c | 29 +++++++++-------------------- 1 file changed, 9 insertions(+), 20 deletions(-)
Quoted 9 lines
kernel/sched/fair.c static bool asym_smt_can_pull_tasks(int dst_cpu, struct sd_lb_stats *sds, · line 8900
Quoted 6 lines
kernel/sched/fair.c +0
-0@@ -8900,12 +8900,10 @@ static bool asym_smt_can_pull_tasks(int dst_cpu, struct sd_lb_stats *sds, struct sched_group *sg)I'm not sure why you change
asym_smt_can_pull_tasks()together with removingSD_ASYM_PACKINGfrom SMT level (patch 5/7)?
In x86 we have SD_ASYM_PACKING at the MC, CLS* and, before my patches, SMT sched domains.
Quoted 7 lines
update_sg_lb_stats()... && env->sd->flags & SD_ASYM_PACKING && .. && sched_asym() ^^^^^^^^^^^^ sched_asym()
Quoted 8 lines
if ((sds->local->flags & SD_SHARE_CPUCAPACITY) || (group->flags & SD_SHARE_CPUCAPACITY)) return asym_smt_can_pull_tasks() ^^^^^^^^^^^^^^^^^^^^^^^^^So x86 won't have a sched domain with
SD_SHARE_CPUCAPACITYandSD_ASYM_PACKINGanymore. Sosched_asym()would callsched_asym_prefer()directly on MC. What do I miss here?
asym_smt_can_pull_tasks() is used above the SMT level *and* when either the local or sg sched groups are composed of CPUs that are SMT siblings.
In fact, asym_smt_can_pull_tasks() can only be called above the SMT level. This is because the flags of a sched_group in a sched_domain are equal to the flags of the child sched_domain. Since SMT is the lowest sched_domain, its groups' flags are 0.
sched_asym() calls sched_asym_prefer() directly if balancing at the SMT level and, at higher domains, if the child domain is not SMT.
This meets the requirement of Power7, where SMT siblings have different priorities; and of x86, where physical cores have different priorities.
Thanks and BR,
Ricardo
* The target of these patches is Intel hybrid processors, on which cluster scheduling is disabled - cabdc3a8475b ("sched,x86: Don't use cluster topology for x86 hybrid CPUs"). Also, I have not observed topologies in which CPUs of the same cluster have different priorities.
We are also looking into re-enabling cluster scheduling on hybrid topologies.
Dietmar Eggemann
reviewerRe: [PATCH v2 1/7] sched/fair: Generalize asym_packing logic for SMT local sched group
Quoted 2 lines · Ricardo Neri
Quoted 1 line · Dietmar Eggemann
On 22/11/2022 21:35, Ricardo Neri wrote:
[...]
Quoted 18 lines
Quoted 2 lines
I'm not sure why you change
asym_smt_can_pull_tasks()together with removingSD_ASYM_PACKINGfrom SMT level (patch 5/7)?In x86 we have
SD_ASYM_PACKINGat the MC, CLS* and, before my patches, SMT sched domains.Quoted 12 lines
update_sg_lb_stats()... && env->sd->flags & SD_ASYM_PACKING && .. && sched_asym() ^^^^^^^^^^^^ sched_asym()if ((sds->local->flags & SD_SHARE_CPUCAPACITY) || (group->flags & SD_SHARE_CPUCAPACITY)) return asym_smt_can_pull_tasks() ^^^^^^^^^^^^^^^^^^^^^^^^^
Quoted 6 lines
Quoted 3 lines
So x86 won't have a sched domain with
SD_SHARE_CPUCAPACITYandSD_ASYM_PACKINGanymore. Sosched_asym()would callsched_asym_prefer()directly on MC. What do I miss here?
asym_smt_can_pull_tasks()is used above the SMT level *and* when either the local or sg sched groups are composed of CPUs that are SMT siblings.
OK.
Quoted 4 lines
In fact,
asym_smt_can_pull_tasks()can only be called above the SMT level. This is because the flags of asched_groupin asched_domainare equal to the flags of the childsched_domain. Since SMT is the lowestsched_domain, its groups' flags are 0.
I see. I forgot about [PATCH v5 0/6] sched/fair: Fix load balancing of SMT siblings with ASYM_PACKING from Sept 21 (specifically [PATCH v5 2/6] sched/topology: Introduce sched_group::flags).
Quoted 2 lines
sched_asym()callssched_asym_prefer()directly if balancing at the SMT level and, at higher domains, if the child domain is not SMT.
OK.
Quoted 10 lines
This meets the requirement of Power7, where SMT siblings have different priorities; and of x86, where physical cores have different priorities.
Thanks and BR,
Ricardo* The target of these patches is Intel hybrid processors, on which cluster scheduling is disabled -
cabdc3a8475b("sched,x86: Don't use cluster topology for x86 hybrid CPUs"). Also, I have not observed topologies in which CPUs of the same cluster have different priorities.
OK.
IMHO, the function header of asym_smt_can_pull_tasks() (3rd and 4th paragraph ... If both @dst_cpu and @sg have SMT siblings and If @sg does not have SMT siblings still reflect the old code layout.
Re: [PATCH v2 1/7] sched/fair: Generalize asym_packing logic for SMT local sched group
Quoted 58 lines · Dietmar Eggemann
Quoted 2 lines · Ricardo Neri
Quoted 1 line · Dietmar Eggemann
On 22/11/2022 21:35, Ricardo Neri wrote:
[...]
Quoted 18 lines
Quoted 2 lines
I'm not sure why you change
asym_smt_can_pull_tasks()together with removingSD_ASYM_PACKINGfrom SMT level (patch 5/7)?In x86 we have
SD_ASYM_PACKINGat the MC, CLS* and, before my patches, SMT sched domains.Quoted 12 lines
update_sg_lb_stats()... && env->sd->flags & SD_ASYM_PACKING && .. && sched_asym() ^^^^^^^^^^^^ sched_asym()if ((sds->local->flags & SD_SHARE_CPUCAPACITY) || (group->flags & SD_SHARE_CPUCAPACITY)) return asym_smt_can_pull_tasks() ^^^^^^^^^^^^^^^^^^^^^^^^^Quoted 6 lines
Quoted 3 lines
So x86 won't have a sched domain with
SD_SHARE_CPUCAPACITYandSD_ASYM_PACKINGanymore. Sosched_asym()would callsched_asym_prefer()directly on MC. What do I miss here?
asym_smt_can_pull_tasks()is used above the SMT level *and* when either the local or sg sched groups are composed of CPUs that are SMT siblings.OK.
Quoted 4 lines
In fact,
asym_smt_can_pull_tasks()can only be called above the SMT level. This is because the flags of asched_groupin asched_domainare equal to the flags of the childsched_domain. Since SMT is the lowestsched_domain, its groups' flags are 0.I see. I forgot about
[PATCH v5 0/6] sched/fair: Fix load balancing of SMT siblings with ASYM_PACKINGfrom Sept 21 (specifically [PATCH v5 2/6] sched/topology: Introducesched_group::flags).Quoted 2 lines
sched_asym()callssched_asym_prefer()directly if balancing at the SMT level and, at higher domains, if the child domain is not SMT.OK.
Quoted 10 lines
This meets the requirement of Power7, where SMT siblings have different priorities; and of x86, where physical cores have different priorities.
Thanks and BR,
Ricardo* The target of these patches is Intel hybrid processors, on which cluster scheduling is disabled -
cabdc3a8475b("sched,x86: Don't use cluster topology for x86 hybrid CPUs"). Also, I have not observed topologies in which CPUs of the same cluster have different priorities.
Quoted 4 lines
OK.
IMHO, the function header of
asym_smt_can_pull_tasks()(3rd and 4th paragraph ...If both @dst_cpu and @sg have SMT siblingsand
Agreed. I changed the behavior of the function. I will update the description.
Quoted 1 line
If @sg does not have SMT siblingsstill reflect the old code layout.
But this behavior did not change. The check covers both SMT and non-SMT cases:
/* * non-SMT @sg can only have 1 busy CPU. We only care SMT @sg * has exactly one busy sibling */ if (sg_busy_cpus == 1 && /* local group is fully idle, SMT and non-SMT. */ !sds->local_stat.sum_nr_running)
Perhaps I can collapse the two paragraphs into one.
Thanks and BR,
Ricardo
Dietmar Eggemann
reviewerRe: [PATCH v2 1/7] sched/fair: Generalize asym_packing logic for SMT local sched group
Quoted 64 lines · Ricardo Neri
Quoted 58 lines · Dietmar Eggemann
Quoted 2 lines · Ricardo Neri
Quoted 1 line · Dietmar Eggemann
On 22/11/2022 21:35, Ricardo Neri wrote:
[...]
Quoted 18 lines
Quoted 2 lines
I'm not sure why you change
asym_smt_can_pull_tasks()together with removingSD_ASYM_PACKINGfrom SMT level (patch 5/7)?In x86 we have
SD_ASYM_PACKINGat the MC, CLS* and, before my patches, SMT sched domains.Quoted 12 lines
update_sg_lb_stats()... && env->sd->flags & SD_ASYM_PACKING && .. && sched_asym() ^^^^^^^^^^^^ sched_asym()if ((sds->local->flags & SD_SHARE_CPUCAPACITY) || (group->flags & SD_SHARE_CPUCAPACITY)) return asym_smt_can_pull_tasks() ^^^^^^^^^^^^^^^^^^^^^^^^^Quoted 6 lines
Quoted 3 lines
So x86 won't have a sched domain with
SD_SHARE_CPUCAPACITYandSD_ASYM_PACKINGanymore. Sosched_asym()would callsched_asym_prefer()directly on MC. What do I miss here?
asym_smt_can_pull_tasks()is used above the SMT level *and* when either the local or sg sched groups are composed of CPUs that are SMT siblings.OK.
Quoted 4 lines
In fact,
asym_smt_can_pull_tasks()can only be called above the SMT level. This is because the flags of asched_groupin asched_domainare equal to the flags of the childsched_domain. Since SMT is the lowestsched_domain, its groups' flags are 0.I see. I forgot about
[PATCH v5 0/6] sched/fair: Fix load balancing of SMT siblings with ASYM_PACKINGfrom Sept 21 (specifically [PATCH v5 2/6] sched/topology: Introducesched_group::flags).Quoted 2 lines
sched_asym()callssched_asym_prefer()directly if balancing at the SMT level and, at higher domains, if the child domain is not SMT.OK.
Quoted 10 lines
This meets the requirement of Power7, where SMT siblings have different priorities; and of x86, where physical cores have different priorities.
Thanks and BR,
Ricardo* The target of these patches is Intel hybrid processors, on which cluster scheduling is disabled -
cabdc3a8475b("sched,x86: Don't use cluster topology for x86 hybrid CPUs"). Also, I have not observed topologies in which CPUs of the same cluster have different priorities.Quoted 4 lines
OK.
IMHO, the function header of
asym_smt_can_pull_tasks()(3rd and 4th paragraph ...If both @dst_cpu and @sg have SMT siblingsand
Quoted 7 lines
Agreed. I changed the behavior of the function. I will update the description.
Quoted 1 line
If @sg does not have SMT siblingsstill reflect the old code layout.But this behavior did not change. The check covers both SMT and non-SMT cases:
The condition to call sched_asym_prefer() seems to have changed slightly though (including the explanation that busy_cpus_delta >= 2 handling should be done by fbg().:
sds->local_stat.sum_nr_running (A) busy_cpus_delta = sg_busy_cpus - local_busy_cpus (B) sg_busy_cpus = sgs->group_weight - sgs->idle_cpus (C)
From ((smt && B == 1) || (!smt && !A)) to (C == 1 && !A)
Quoted 10 lines
/* * non-SMT @sg can only have 1 busy CPU. We only care SMT @sg * has exactly one busy sibling */ if (sg_busy_cpus == 1 && /* local group is fully idle, SMT and non-SMT. */ !sds->local_stat.sum_nr_running)Perhaps I can collapse the two paragraphs into one.
Sounds good to me.
[...]
Re: [PATCH v2 1/7] sched/fair: Generalize asym_packing logic for SMT local sched group
Quoted 77 lines · Dietmar Eggemann
Quoted 64 lines · Ricardo Neri
Quoted 58 lines · Dietmar Eggemann
Quoted 2 lines · Ricardo Neri
Quoted 1 line · Dietmar Eggemann
On 22/11/2022 21:35, Ricardo Neri wrote:
[...]
Quoted 18 lines
Quoted 2 lines
I'm not sure why you change
asym_smt_can_pull_tasks()together with removingSD_ASYM_PACKINGfrom SMT level (patch 5/7)?In x86 we have
SD_ASYM_PACKINGat the MC, CLS* and, before my patches, SMT sched domains.Quoted 12 lines
update_sg_lb_stats()... && env->sd->flags & SD_ASYM_PACKING && .. && sched_asym() ^^^^^^^^^^^^ sched_asym()if ((sds->local->flags & SD_SHARE_CPUCAPACITY) || (group->flags & SD_SHARE_CPUCAPACITY)) return asym_smt_can_pull_tasks() ^^^^^^^^^^^^^^^^^^^^^^^^^Quoted 6 lines
Quoted 3 lines
So x86 won't have a sched domain with
SD_SHARE_CPUCAPACITYandSD_ASYM_PACKINGanymore. Sosched_asym()would callsched_asym_prefer()directly on MC. What do I miss here?
asym_smt_can_pull_tasks()is used above the SMT level *and* when either the local or sg sched groups are composed of CPUs that are SMT siblings.OK.
Quoted 4 lines
In fact,
asym_smt_can_pull_tasks()can only be called above the SMT level. This is because the flags of asched_groupin asched_domainare equal to the flags of the childsched_domain. Since SMT is the lowestsched_domain, its groups' flags are 0.I see. I forgot about
[PATCH v5 0/6] sched/fair: Fix load balancing of SMT siblings with ASYM_PACKINGfrom Sept 21 (specifically [PATCH v5 2/6] sched/topology: Introducesched_group::flags).Quoted 2 lines
sched_asym()callssched_asym_prefer()directly if balancing at the SMT level and, at higher domains, if the child domain is not SMT.OK.
Quoted 10 lines
This meets the requirement of Power7, where SMT siblings have different priorities; and of x86, where physical cores have different priorities.
Thanks and BR,
Ricardo* The target of these patches is Intel hybrid processors, on which cluster scheduling is disabled -
cabdc3a8475b("sched,x86: Don't use cluster topology for x86 hybrid CPUs"). Also, I have not observed topologies in which CPUs of the same cluster have different priorities.Quoted 4 lines
OK.
IMHO, the function header of
asym_smt_can_pull_tasks()(3rd and 4th paragraph ...If both @dst_cpu and @sg have SMT siblingsandQuoted 7 lines
Agreed. I changed the behavior of the function. I will update the description.
Quoted 1 line
If @sg does not have SMT siblingsstill reflect the old code layout.But this behavior did not change. The check covers both SMT and non-SMT cases:
The condition to call
sched_asym_prefer()seems to have changed slightly though (including the explanation thatbusy_cpus_delta>= 2 handling should be done byfbg().:
Quoted 5 lines
sds->local_stat.sum_nr_running(A)busy_cpus_delta=sg_busy_cpus-local_busy_cpus(B)sg_busy_cpus=sgs->group_weight-sgs->idle_cpus(C)From ((smt && B == 1) || (!smt && !A)) to (C == 1 && !A)
I agree that ((smt && B == 1) did change and I will update the comment.
My point is that (!smt && !A) is equivalent to (C == 1 && !A) if @sg has only one CPU and is busy. The fourth paragraph still stands.
Quoted 10 lines
Quoted 10 lines
/* * non-SMT @sg can only have 1 busy CPU. We only care SMT @sg * has exactly one busy sibling */ if (sg_busy_cpus == 1 && /* local group is fully idle, SMT and non-SMT. */ !sds->local_stat.sum_nr_running)
Quoted 3 lines
Quoted 1 line
Perhaps I can collapse the two paragraphs into one.
Sounds good to me.
Will do.
Thanks and BR,
Ricardo
Valentin Schneider
reviewerRe: [PATCH v2 1/7] sched/fair: Generalize asym_packing logic for SMT local sched group
In reply to Ricardo NeriQuoted 23 lines · Ricardo Neri
(quoted diff) +2
-16@@ -8926,25 +8924,16 @@ static bool asym_smt_can_pull_tasks(int dst_cpu, struct sd_lb_stats *sds, return sched_asym_prefer(dst_cpu, sg->asym_prefer_cpu); } - /* @dst_cpu has SMT siblings. */ - - if (sg_is_smt) { - int local_busy_cpus = sds->local->group_weight - - sds->local_stat.idle_cpus; - int busy_cpus_delta = sg_busy_cpus - local_busy_cpus; - - if (busy_cpus_delta == 1) - return sched_asym_prefer(dst_cpu, sg->asym_prefer_cpu); - - return false; - } - /* - * @sg does not have SMT siblings. Ensure that @sds::local does not end - * up with more than one busy SMT sibling and only pull tasks if there - * are not busy CPUs (i.e., no CPU has running tasks). + * @dst_cpu has SMT siblings. Do asym_packing load balancing only if + * all its siblings are idle (moving tasks between physical cores in
Review static bool asym_smt_can_pull_tasks(int dst_cpu, struct sd_lb_stats *sds, · line 8938
Quoted 10 lines
(quoted diff) +6
-1+ * which some SMT siblings are busy results in the same throughput). + * + * If the difference in the number of busy CPUs is two or more, let + * find_busiest_group() take care of it. We only care if @sg has + * exactly one busy CPU. This covers SMT and non-SMT sched groups. */ - if (!sds->local_stat.sum_nr_running) + if (sg_busy_cpus == 1 && !sds->local_stat.sum_nr_running) return sched_asym_prefer(dst_cpu, sg->asym_prefer_cpu);
Some of this is new to me - I had missed the original series introducing this. However ISTM that this is conflating two concepts: SMT occupancy balancing, and asym packing.
Take the !local_is_smt :: sg_busy_cpus >= 2 :: return true; path. It does not involve asym packing priorities at all. This can end up in an ASYM_PACKING load balance, which per calculate_imbalance() tries to move *all* tasks to the higher priority CPU - in the case of SMT balancing, we don't want to totally empty the core, just its siblings.
Is there an ITMT/big.LITTLE (or however x86 calls it) case that invalidates the above?
Say, what's not sufficient with the below? AFAICT the only thing that isn't covered is the sg_busy_cpus >= 2 thing, but IMO that's SMT balancing, not asym packing - if the current calculate_imbalance() doesn't do it, it should be fixed to do it. Looking at the
local->group_type == group_has_spare
case, it looks like it should DTRT.
---
kernel/sched/fair.c +8 -5
@@ -9176,12 +9176,15 @@ static inline bool sched_asym(struct lb_env *env, struct sd_lb_stats *sds, struct sg_lb_stats *sgs, struct sched_group *group) { - /* Only do SMT checks if either local or candidate have SMT siblings */ - if ((sds->local->flags & SD_SHARE_CPUCAPACITY) || - (group->flags & SD_SHARE_CPUCAPACITY)) - return asym_smt_can_pull_tasks(env->dst_cpu, sds, sgs, group); + /* + * For SMT, env->idle != CPU_NOT_IDLE isn't sufficient, we need to make + * sure the whole core is idle. + */ + if (((sds->local->flags & SD_SHARE_CPUCAPACITY) || + (group->flags & SD_SHARE_CPUCAPACITY)) && + !test_idle_cores(env->dst_cpu)) + return false; -/* Neither env::dst_cpu nor group::asym_prefer_cpu have SMT siblings. */ return sched_asym_prefer(env->dst_cpu, group->asym_prefer_cpu, false); }
Re: [PATCH v2 1/7] sched/fair: Generalize asym_packing logic for SMT local sched group
Quoted 39 lines · Valentin Schneider
Quoted 23 lines · Ricardo Neri
(quoted diff) +2
-16@@ -8926,25 +8924,16 @@ static bool asym_smt_can_pull_tasks(int dst_cpu, struct sd_lb_stats *sds, return sched_asym_prefer(dst_cpu, sg->asym_prefer_cpu); } - /* @dst_cpu has SMT siblings. */ - - if (sg_is_smt) { - int local_busy_cpus = sds->local->group_weight - - sds->local_stat.idle_cpus; - int busy_cpus_delta = sg_busy_cpus - local_busy_cpus; - - if (busy_cpus_delta == 1) - return sched_asym_prefer(dst_cpu, sg->asym_prefer_cpu); - - return false; - } - /* - * @sg does not have SMT siblings. Ensure that @sds::local does not end - * up with more than one busy SMT sibling and only pull tasks if there - * are not busy CPUs (i.e., no CPU has running tasks). + * @dst_cpu has SMT siblings. Do asym_packing load balancing only if + * all its siblings are idle (moving tasks between physical cores inReview static bool asym_smt_can_pull_tasks(int dst_cpu, struct sd_lb_stats *sds, · line 8938
Quoted 10 lines
(quoted diff) +6
-1+ * which some SMT siblings are busy results in the same throughput). + * + * If the difference in the number of busy CPUs is two or more, let + * find_busiest_group() take care of it. We only care if @sg has + * exactly one busy CPU. This covers SMT and non-SMT sched groups. */ - if (!sds->local_stat.sum_nr_running) + if (sg_busy_cpus == 1 && !sds->local_stat.sum_nr_running) return sched_asym_prefer(dst_cpu, sg->asym_prefer_cpu);Some of this is new to me - I had missed the original series introducing this. However ISTM that this is conflating two concepts: SMT occupancy balancing, and asym packing.
Quoted 3 lines
Take the !local_is_smt ::
sg_busy_cpus>= 2 :: return true; path. It does not involve asym packing priorities at all. This can end up in anASYM_PACKINGload balance,
Yes, this the desired result: an idle low-priority CPU should help a high- priority core with more than one busy SMT sibling. But yes, it does not relate to priorities and can be implemented differently.
Quoted 3 lines
which per
calculate_imbalance()tries to move
*all* tasks to the higher priority CPU - in the case of SMT balancing, we don't want to totally empty the core, just its siblings.
But it will not empty the core, only one of its SMT siblings. A single sibling will be selected in find_busiest_queue(). The other siblings will be unaffected.
Quoted 3 lines
Is there an ITMT/big.LITTLE (or however x86 calls it) case that invalidates the above?
Please see below.
Quoted 5 lines
Say, what's not sufficient with the below? AFAICT the only thing that isn't covered is the
sg_busy_cpus>= 2 thing, but IMO that's SMT balancing, not asym packing - if the currentcalculate_imbalance()doesn't do it, it should be fixed to do it.
Agreed.
Quoted 5 lines
Looking at the
local->group_type==group_has_sparecase, it looks like it should DTRT.
I had tried (and failed) to have find_busiest_group() handle the !local_is_smt :: sg_busy_cpus >= 2 case. Now I think I made it work.
When the busiest group is not overloaded, find_busiest_group() and local->group = group_has_spare during an idle load balance events the number of *idle* CPUs. However, this does not work if the local and busiest groups have different weights. In SMT2, for instance, if busiest has 2 busy CPUs (i.e., 0 idle CPUs) and the destination CPU is idle, the difference in the number of idle CPUs is 1. find_busiest_group() will incorrectly goto out_balanced.
This issue very visible in Intel hybrid processors because the big cores have SMT but the small cores do not. It can, however, be reproduced in non- hybrid processors by offlining the SMT siblings of some cores.
The problem can be fixed by instead balancing the number of *busy* CPUs, which is what in general we want, IMO. (When two groups have the same weight, it is equivalent to balancing the number of idle CPUs).
This patch worked for me:
@@ -9787,14 +9787,18 @@ static inline void calculate_imbalance(struct lb_env *env, struct sd_lb_stats *s
lsub_positive(&nr_diff, local->sum_nr_running);
env->imbalance = nr_diff;
} else {
+ unsigned int busiest_busy_cpus, local_busy_cpus;
+
+ busiest_busy_cpus = busiest->group_weight - busiest->idle_cpus;
+ local_busy_cpus = local->group_weight - local->idle_cpus;
/*
* If there is no overload, we just want to even the number of
- * idle cpus.
+ * busy cpus.
*/
env->migration_type = migrate_task;
- env->imbalance = max_t(long, 0,
- (local->idle_cpus - busiest->idle_cpus));
+ env->imbalance = max_t(long, 0 ,
+ (busiest_busy_cpus - local_busy_cpus));
}
#ifdef CONFIG_NUMA
@@ -9981,18 +9985,24 @@ static struct sched_group *find_busiest_group(struct lb_env *env)
*/
goto out_balanced;
- if (busiest->group_weight > 1 &&
- local->idle_cpus <= (busiest->idle_cpus + 1))
- /*
- * If the busiest group is not overloaded
- * and there is no imbalance between this and busiest
- * group wrt idle CPUs, it is balanced. The imbalance
- * becomes significant if the diff is greater than 1
- * otherwise we might end up to just move the imbalance
- * on another group. Of course this applies only if
- * there is more than 1 CPU per group.
- */
- goto out_balanced;
+ if (busiest->group_weight > 1) {
+ unsigned int local_busy_cpus, busiest_busy_cpus;
+
+ local_busy_cpus = local->group_weight - local->idle_cpus;
+ busiest_busy_cpus = busiest->group_weight - busiest->idle_cpus;
+
+ if (busiest_busy_cpus <= local_busy_cpus + 1)
+ /*
+ * If the busiest group is not overloaded
+ * and there is no imbalance between this and busiest
+ * group wrt busy CPUs, it is balanced. The imbalance
+ * becomes significant if the diff is greater than 1
+ * otherwise we might end up to just move the imbalance
+ * on another group. Of course this applies only if
+ * there is more than 1 CPU per group.
+ */
+ goto out_balanced;
+ }
if (busiest->sum_h_nr_running == 1)
/*
With this I can remove the sg_busy_cpus >=2 thing from asym_smt_can_pull_tasks().
Quoted 11 lines
---
kernel/sched/fair.c +0
-1@@ -9176,12 +9176,15 @@ static inline bool sched_asym(struct lb_env *env, struct sd_lb_stats *sds, struct sg_lb_stats *sgs, struct sched_group *group) { -/* Only do SMT checks if either local or candidate have SMT siblings */
kernel/sched/fair.c static inline bool · line 9185
Quoted 10 lines
kernel/sched/fair.c +7
-3-if ((sds->local->flags &SD_SHARE_CPUCAPACITY ) || - (group->flags &SD_SHARE_CPUCAPACITY )) -return asym_smt_can_pull_tasks (env->dst_cpu, sds, sgs, group); +/* +* For SMT, env->idle != CPU_NOT_IDLE isn't sufficient, we need to make +* sure the whole core is idle. +*/ +if (((sds->local->flags &SD_SHARE_CPUCAPACITY ) || + (group->flags &SD_SHARE_CPUCAPACITY )) && + !test_idle_cores (env->dst_cpu))
But test_idle_cores() tests for *any* idle core in the highest domain with the SD_SHARE_PKG_RESOURCES flag. Here we are only interested in the SMT siblings of env->dst_cpu. If is_core_idle(env->dst_cpu) is used, then I agree with the change.
Thanks and BR,
Ricardo
Valentin Schneider
reviewerRe: [PATCH v2 1/7] sched/fair: Generalize asym_packing logic for SMT local sched group
Quoted 13 lines · Ricardo Neri
Quoted 7 lines · Valentin Schneider
Some of this is new to me - I had missed the original series introducing this. However ISTM that this is conflating two concepts: SMT occupancy balancing, and asym packing.
Take the !local_is_smt ::
sg_busy_cpus>= 2 :: return true; path. It does not involve asym packing priorities at all. This can end up in anASYM_PACKINGload balance,Yes, this the desired result: an idle low-priority CPU should help a high- priority core with more than one busy SMT sibling. But yes, it does not relate to priorities and can be implemented differently.
Quoted 8 lines
Quoted 3 lines
which per
calculate_imbalance()tries to move
*all* tasks to the higher priority CPU - in the case of SMT balancing, we don't want to totally empty the core, just its siblings.But it will not empty the core, only one of its SMT siblings. A single sibling will be selected in
find_busiest_queue(). The other siblings will be unaffected.
Right
Quoted 31 lines
Quoted 3 lines
Is there an ITMT/big.LITTLE (or however x86 calls it) case that invalidates the above?
Please see below.
Quoted 5 lines
Say, what's not sufficient with the below? AFAICT the only thing that isn't covered is the
sg_busy_cpus>= 2 thing, but IMO that's SMT balancing, not asym packing - if the currentcalculate_imbalance()doesn't do it, it should be fixed to do it.Agreed.
Quoted 5 lines
Looking at the
local->group_type==group_has_sparecase, it looks like it should DTRT.
I had tried (and failed) to have
find_busiest_group()handle the !local_is_smt ::sg_busy_cpus>= 2 case. Now I think I made it work.When the busiest group is not overloaded,
find_busiest_group()andlocal->group=group_has_spareduring an idle load balance events the number of *idle* CPUs. However, this does not work if the local and busiest groups have different weights. In SMT2, for instance, if busiest has 2 busy CPUs (i.e., 0 idle CPUs) and the destination CPU is idle, the difference in the number of idle CPUs is 1.find_busiest_group()will incorrectly gotoout_balanced.
Quoted 4 lines
This issue very visible in Intel hybrid processors because the big cores have SMT but the small cores do not. It can, however, be reproduced in non- hybrid processors by offlining the SMT siblings of some cores.
I think I follow. If we're comparing two groups each spanning an SMT2 core, then
busiest->group_weight > 1 && local->idle_cpus <= (busiest->idle_cpus + 1)
is false if local is fully idle and busiest fully busy, but if local becomes a non-SMT core, then that's true and we goto out_balanced.
With that said, shouldn't SD_PREFER_SIBLING help here? cf.
if (sds.prefer_sibling && local->group_type == group_has_spare && busiest->sum_nr_running > local->sum_nr_running + 1)
It should be set on any topology level below the NUMA ones, we do remove it on SD_ASYM_CPUCAPACITY levels because this used to interfere with misfit balancing (it would override the group_type), things are a bit different since Vincent's rewrite of load_balance() but I think we still want it off there. I would expect it to be set in your system, though whether this is playing nice with the asymmetry is another matter :-)
Quoted 86 lines
The problem can be fixed by instead balancing the number of *busy* CPUs, which is what in general we want, IMO. (When two groups have the same weight, it is equivalent to balancing the number of idle CPUs).
This patch worked for me:
(quoted diff) +25
-15@@ -9787,14 +9787,18 @@ static inline void calculate_imbalance(struct lb_env *env, struct sd_lb_stats *s lsub_positive(&nr_diff, local->sum_nr_running); env->imbalance = nr_diff; } else { + unsigned int busiest_busy_cpus, local_busy_cpus; + + busiest_busy_cpus = busiest->group_weight - busiest->idle_cpus; + local_busy_cpus = local->group_weight - local->idle_cpus; /* * If there is no overload, we just want to even the number of - * idle cpus. + * busy cpus. */ env->migration_type = migrate_task; - env->imbalance = max_t(long, 0, - (local->idle_cpus - busiest->idle_cpus)); + env->imbalance = max_t(long, 0 , + (busiest_busy_cpus - local_busy_cpus)); } #ifdef CONFIG_NUMA@@ -9981,18 +9985,24 @@ static struct sched_group *find_busiest_group(struct lb_env *env) */ goto out_balanced; - if (busiest->group_weight > 1 && - local->idle_cpus <= (busiest->idle_cpus + 1)) - /* - * If the busiest group is not overloaded - * and there is no imbalance between this and busiest - * group wrt idle CPUs, it is balanced. The imbalance - * becomes significant if the diff is greater than 1 - * otherwise we might end up to just move the imbalance - * on another group. Of course this applies only if - * there is more than 1 CPU per group. - */ - goto out_balanced; + if (busiest->group_weight > 1) { + unsigned int local_busy_cpus, busiest_busy_cpus; + + local_busy_cpus = local->group_weight - local->idle_cpus; + busiest_busy_cpus = busiest->group_weight - busiest->idle_cpus; + + if (busiest_busy_cpus <= local_busy_cpus + 1) + /* + * If the busiest group is not overloaded + * and there is no imbalance between this and busiest + * group wrt busy CPUs, it is balanced. The imbalance + * becomes significant if the diff is greater than 1 + * otherwise we might end up to just move the imbalance + * on another group. Of course this applies only if + * there is more than 1 CPU per group. + */ + goto out_balanced; + } if (busiest->sum_h_nr_running == 1) /*With this I can remove the
sg_busy_cpus>=2 thing fromasym_smt_can_pull_tasks().Quoted 18 lines
---
kernel/sched/fair.c +4
-4@@ -9176,12 +9176,15 @@ static inline bool sched_asym(struct lb_env *env, struct sd_lb_stats *sds, struct sg_lb_stats *sgs, struct sched_group *group) { - /* Only do SMT checks if either local or candidate have SMT siblings */ - if ((sds->local->flags & SD_SHARE_CPUCAPACITY) || - (group->flags & SD_SHARE_CPUCAPACITY)) - return asym_smt_can_pull_tasks(env->dst_cpu, sds, sgs, group); +/* +* For SMT, env->idle != CPU_NOT_IDLE isn't sufficient, we need to make +* sure the whole core is idle. +*/
Quoted 9 lines
kernel/sched/fair.c static inline bool · line 9185
Quoted 3 lines
kernel/sched/fair.c +3
-0+if (((sds->local->flags &SD_SHARE_CPUCAPACITY ) || + (group->flags &SD_SHARE_CPUCAPACITY )) && + !test_idle_cores (env->dst_cpu))But
test_idle_cores()tests for *any* idle core in the highest domain with theSD_SHARE_PKG_RESOURCESflag. Here we are only interested in the SMT siblings ofenv->dst_cpu. Ifis_core_idle(env->dst_cpu)is used, then I agree with the change.
Right, I had gotten confused with test_idle_cores()
Quoted 2 lines
Thanks and BR,
Ricardo
Re: [PATCH v2 1/7] sched/fair: Generalize asym_packing logic for SMT local sched group
Quoted 61 lines · Valentin Schneider
Quoted 13 lines · Ricardo Neri
Quoted 7 lines · Valentin Schneider
Some of this is new to me - I had missed the original series introducing this. However ISTM that this is conflating two concepts: SMT occupancy balancing, and asym packing.
Take the !local_is_smt ::
sg_busy_cpus>= 2 :: return true; path. It does not involve asym packing priorities at all. This can end up in anASYM_PACKINGload balance,Yes, this the desired result: an idle low-priority CPU should help a high- priority core with more than one busy SMT sibling. But yes, it does not relate to priorities and can be implemented differently.
Quoted 8 lines
Quoted 3 lines
which per
calculate_imbalance()tries to move
*all* tasks to the higher priority CPU - in the case of SMT balancing, we don't want to totally empty the core, just its siblings.But it will not empty the core, only one of its SMT siblings. A single sibling will be selected in
find_busiest_queue(). The other siblings will be unaffected.Right
Quoted 31 lines
Quoted 3 lines
Is there an ITMT/big.LITTLE (or however x86 calls it) case that invalidates the above?
Please see below.
Quoted 5 lines
Say, what's not sufficient with the below? AFAICT the only thing that isn't covered is the
sg_busy_cpus>= 2 thing, but IMO that's SMT balancing, not asym packing - if the currentcalculate_imbalance()doesn't do it, it should be fixed to do it.Agreed.
Quoted 5 lines
Looking at the
local->group_type==group_has_sparecase, it looks like it should DTRT.
I had tried (and failed) to have
find_busiest_group()handle the !local_is_smt ::sg_busy_cpus>= 2 case. Now I think I made it work.When the busiest group is not overloaded,
find_busiest_group()andlocal->group=group_has_spareduring an idle load balance events the number of *idle* CPUs. However, this does not work if the local and busiest groups have different weights. In SMT2, for instance, if busiest has 2 busy CPUs (i.e., 0 idle CPUs) and the destination CPU is idle, the difference in the number of idle CPUs is 1.find_busiest_group()will incorrectly gotoout_balanced.Quoted 4 lines
This issue very visible in Intel hybrid processors because the big cores have SMT but the small cores do not. It can, however, be reproduced in non- hybrid processors by offlining the SMT siblings of some cores.
Quoted 7 lines
I think I follow. If we're comparing two groups each spanning an SMT2 core, then
busiest->group_weight > 1 && local->idle_cpus <= (busiest->idle_cpus + 1)is false if local is fully idle and busiest fully busy, but if local becomes a non-SMT core, then that's true and we goto
out_balanced.
Exactly right.
Quoted 6 lines
With that said, shouldn't
SD_PREFER_SIBLINGhelp here? cf.if (sds.prefer_sibling && local->group_type == group_has_spare && busiest->sum_nr_running > local->sum_nr_running + 1)
It does not help because sds.prefer_sibling is false: an non-SMT core is looking into offloading a fully_busy SMT core at the "MC" domain. sds.prefer_sibling is set in update_sd_lb_stats() if the sched domain's child has the SD_PREFER_SIBLING flag. Since the destination CPU is the non-SMT core, there is no child.
Quoted 6 lines
It should be set on any topology level below the NUMA ones, we do remove it on
SD_ASYM_CPUCAPACITYlevels because this used to interfere with misfit balancing (it would override thegroup_type), things are a bit different since Vincent's rewrite ofload_balance()but I think we still want it off there.
I see in find_busiest_group() that group_misfit_task is evaluated before sds.prefer_sibling.
Quoted 2 lines
I would expect it to be set in your system, though whether this is playing nice with the asymmetry is another matter :-)
I recall a few instances of SD_PREFER_SIBLING causing trouble me, but I need to investigate more.
Thanks and BR,
Ricardo
Re: [PATCH v2 1/7] sched/fair: Generalize asym_packing logic for SMT local sched group
Quoted 76 lines · Ricardo Neri
Quoted 61 lines · Valentin Schneider
Quoted 13 lines · Ricardo Neri
Quoted 7 lines · Valentin Schneider
Some of this is new to me - I had missed the original series introducing this. However ISTM that this is conflating two concepts: SMT occupancy balancing, and asym packing.
Take the !local_is_smt ::
sg_busy_cpus>= 2 :: return true; path. It does not involve asym packing priorities at all. This can end up in anASYM_PACKINGload balance,Yes, this the desired result: an idle low-priority CPU should help a high- priority core with more than one busy SMT sibling. But yes, it does not relate to priorities and can be implemented differently.
Quoted 8 lines
Quoted 3 lines
which per
calculate_imbalance()tries to move
*all* tasks to the higher priority CPU - in the case of SMT balancing, we don't want to totally empty the core, just its siblings.But it will not empty the core, only one of its SMT siblings. A single sibling will be selected in
find_busiest_queue(). The other siblings will be unaffected.Right
Quoted 31 lines
Quoted 3 lines
Is there an ITMT/big.LITTLE (or however x86 calls it) case that invalidates the above?
Please see below.
Quoted 5 lines
Say, what's not sufficient with the below? AFAICT the only thing that isn't covered is the
sg_busy_cpus>= 2 thing, but IMO that's SMT balancing, not asym packing - if the currentcalculate_imbalance()doesn't do it, it should be fixed to do it.Agreed.
Quoted 5 lines
Looking at the
local->group_type==group_has_sparecase, it looks like it should DTRT.
I had tried (and failed) to have
find_busiest_group()handle the !local_is_smt ::sg_busy_cpus>= 2 case. Now I think I made it work.When the busiest group is not overloaded,
find_busiest_group()andlocal->group=group_has_spareduring an idle load balance events the number of *idle* CPUs. However, this does not work if the local and busiest groups have different weights. In SMT2, for instance, if busiest has 2 busy CPUs (i.e., 0 idle CPUs) and the destination CPU is idle, the difference in the number of idle CPUs is 1.find_busiest_group()will incorrectly gotoout_balanced.Quoted 4 lines
This issue very visible in Intel hybrid processors because the big cores have SMT but the small cores do not. It can, however, be reproduced in non- hybrid processors by offlining the SMT siblings of some cores.
Quoted 7 lines
I think I follow. If we're comparing two groups each spanning an SMT2 core, then
busiest->group_weight > 1 && local->idle_cpus <= (busiest->idle_cpus + 1)is false if local is fully idle and busiest fully busy, but if local becomes a non-SMT core, then that's true and we goto
out_balanced.Exactly right.
Quoted 4 lines
With that said, shouldn't
SD_PREFER_SIBLINGhelp here? cf.
Quoted 15 lines
Quoted 2 lines
if (sds.prefer_sibling && local->group_type == group_has_spare && busiest->sum_nr_running > local->sum_nr_running + 1)It does not help because sds.prefer_sibling is false: an non-SMT core is looking into offloading a
fully_busySMT core at the "MC" domain. sds.prefer_sibling is set inupdate_sd_lb_stats()if the sched domain's child has theSD_PREFER_SIBLINGflag. Since the destination CPU is the non-SMT core, there is no child.Quoted 6 lines
It should be set on any topology level below the NUMA ones, we do remove it on
SD_ASYM_CPUCAPACITYlevels because this used to interfere with misfit balancing (it would override thegroup_type), things are a bit different since Vincent's rewrite ofload_balance()but I think we still want it off there.
Your comment got me thinking. Whose child sched domain wants prefer_sibling? It sounds to me that is busiest's. I could not think of any reason of *having* to use the flags of the local group.
We can use the flags of the sched group (as per 16d364ba6ef2 ("sched/topology:
Introduce sched_group::flags"), these are the flags of the child domain).
The patch below works for me and I don't have to even the number of busy CPUs. It should not interfere with misfit balancing either:
kernel/sched/fair.c +6 -2
@@ -9752,8 +9752,12 @@ static inline void update_sd_lb_stats(struct lb_env *env, struct sd_lb_stats *sd sg = sg->next; }while (sg != env->sd->groups); -/* Tag domain that child domain prefers tasks go to siblings first */ - sds->prefer_sibling = child && child->flags &SD_PREFER_SIBLING ; +/* +* Tag domain that child domain prefers tasks go to siblings first. +* A sched group has the flags of the child domain, if any. +*/ +if (sds->busiest) + sds->prefer_sibling = sds->busiest->flags &SD_PREFER_SIBLING ;if (env->sd->flags &SD_NUMA )
Thanks and BR,
Ricardo
Valentin Schneider
reviewerRe: [PATCH v2 1/7] sched/fair: Generalize asym_packing logic for SMT local sched group
Quoted 11 lines · Ricardo Neri
Quoted 6 lines
Quoted 6 lines
It should be set on any topology level below the NUMA ones, we do remove it on
SD_ASYM_CPUCAPACITYlevels because this used to interfere with misfit balancing (it would override thegroup_type), things are a bit different since Vincent's rewrite ofload_balance()but I think we still want it off there.Your comment got me thinking. Whose child sched domain wants
prefer_sibling? It sounds to me that is busiest's. I could not think of any reason of *having* to use the flags of the local group.
Hm, given that on systems without SD_ASYM_CPUCAPACITY, SD_PREFER_SIBLING is set all the way from SMT up to the last !NUMA domain, should we just get rid of the child/parent weirdness of SD_PREFER_SIBLING and stick with the flags we are given at the level we're balancing at?
i.e.
sds->prefer_sibling = env->sd & SD_PREFER_SIBLING;
Unless I'm reading this wrong, this also eliminates the effect of SD_PREFER_SIBLING on the first NUMA level - DIE level has SD_PREFER_SIBLING set, but we don't necessarily want to evenly spread things out when accross NUMA nodes.
Quoted 6 lines
We can use the flags of the sched group (as per
16d364ba6ef2("sched/topology:
Introducesched_group::flags"), these are the flags of the child domain).The patch below works for me and I don't have to even the number of busy CPUs. It should not interfere with misfit balancing either:
We remove that flag on systems where misfit balancing happens anyway, so that's safe vs. SD_PREFER_SIBLING.
Re: [PATCH v2 1/7] sched/fair: Generalize asym_packing logic for SMT local sched group
Quoted 13 lines · Valentin Schneider
Quoted 11 lines · Ricardo Neri
Quoted 6 lines
Quoted 6 lines
It should be set on any topology level below the NUMA ones, we do remove it on
SD_ASYM_CPUCAPACITYlevels because this used to interfere with misfit balancing (it would override thegroup_type), things are a bit different since Vincent's rewrite ofload_balance()but I think we still want it off there.Your comment got me thinking. Whose child sched domain wants
prefer_sibling? It sounds to me that is busiest's. I could not think of any reason of *having* to use the flags of the local group.
Quoted 8 lines
Hm, given that on systems without
SD_ASYM_CPUCAPACITY,SD_PREFER_SIBLINGis set all the way from SMT up to the last !NUMA domain, should we just get rid of the child/parent weirdness ofSD_PREFER_SIBLINGand stick with the flags we are given at the level we're balancing at?i.e.
sds->prefer_sibling = env->sd & SD_PREFER_SIBLING;
Agreed. This would also make the code easier to understand. It should not change the current behavior either; except (i.e., fix) for the busiest->group_weight = 2 vs local->group_weight = 1 I raised.
Quoted 5 lines
Unless I'm reading this wrong, this also eliminates the effect of
SD_PREFER_SIBLINGon the first NUMA level - DIE level hasSD_PREFER_SIBLINGset, but we don't necessarily want to evenly spread things out when accross NUMA nodes.
Agreed.
Quoted 11 lines
Quoted 8 lines
We can use the flags of the sched group (as per
16d364ba6ef2("sched/topology:
Introducesched_group::flags"), these are the flags of the child domain).The patch below works for me and I don't have to even the number of busy CPUs. It should not interfere with misfit balancing either:
We remove that flag on systems where misfit balancing happens anyway, so that's safe vs.
SD_PREFER_SIBLING.
Then all looks good with your suggestion. I'll include a patch in my series.
Thanks and BR,
Ricardo
PATCH v2 2/7 sched: Prepare sched_asym_prefer() to handle idle state of SMT siblings
In reply to Ricardo NeriThe throughput of an SMT sibling decreases if one or more of its siblings are also busy. Idle, lower-priority cores can help. Thus, it is necessary to consider the idle state of the SMT siblings of CPUs when selecting by priority.
In some cases, sched_asym_prefer() does not care about the idle state (when building sched domains or looking at the priority of the preferred CPU in a sched group).
Add a new parameter to check the state of the SMT siblings of a CPU when applicable.
While here, remove a spurious newline.
---
Changes since v1:
* Introduced this patch
--- kernel/sched/fair.c | 17 ++++++++++------- kernel/sched/sched.h | 8 ++++++-- kernel/sched/topology.c | 6 +++++- 3 files changed, 21 insertions(+), 10 deletions(-)
kernel/sched/fair.c +10 -7
@@ -8921,7 +8921,7 @@ static bool asym_smt_can_pull_tasks(int dst_cpu, struct sd_lb_stats *sds,* can help if it has higher priority and is idle (i.e., * it has no running tasks). */ -return sched_asym_prefer (dst_cpu, sg->asym_prefer_cpu); +return sched_asym_prefer (dst_cpu, sg->asym_prefer_cpu, false); }/*
@@ -8934,7 +8934,7 @@ static bool asym_smt_can_pull_tasks(int dst_cpu, struct sd_lb_stats *sds,* exactly one busy CPU. This covers SMT and non-SMT sched groups. */ if (sg_busy_cpus ==1 && !sds->local_stat.sum_nr_running) -return sched_asym_prefer (dst_cpu, sg->asym_prefer_cpu); +return sched_asym_prefer (dst_cpu, sg->asym_prefer_cpu, false);return false;#else
@@ -8952,7 +8952,8 @@ sched_asym(struct lb_env *env, struct sd_lb_stats *sds, struct sg_lb_stats *sgs (group->flags & SD_SHARE_CPUCAPACITY))return asym_smt_can_pull_tasks (env->dst_cpu, sds, sgs, group); -return sched_asym_prefer (env->dst_cpu, group->asym_prefer_cpu); +/* Neither env::dst_cpu nor group::asym_prefer_cpu have SMT siblings. */ +return sched_asym_prefer (env->dst_cpu, group->asym_prefer_cpu, false); } static inline bool
@@ -9118,7 +9119,9 @@ static bool update_sd_pick_busiest(struct lb_env *env,case group_asym_packing:/* Prefer to move from lowest priority CPU's work */ -if (sched_asym_prefer (sg->asym_prefer_cpu, sds->busiest->asym_prefer_cpu)) +if (sched_asym_prefer (sg->asym_prefer_cpu, + sds->busiest->asym_prefer_cpu, + false))return false;break ;
@@ -10060,7 +10063,7 @@ static struct rq *find_busiest_queue(struct lb_env *env,/* Make sure we only pull tasks from a CPU of lower priority */ if ((env->sd->flags &SD_ASYM_PACKING ) && -sched_asym_prefer (i, env->dst_cpu) && +sched_asym_prefer (i, env->dst_cpu, true) && nr_running ==1 )continue ;
@@ -10153,7 +10156,7 @@ asym_active_balance(struct lb_env *env)* highest priority CPUs. */ return env->idle !=CPU_NOT_IDLE && (env->sd->flags &SD_ASYM_PACKING ) && -sched_asym_prefer (env->dst_cpu, env->src_cpu); +sched_asym_prefer (env->dst_cpu, env->src_cpu, true); } static inline bool
@@ -10889,7 +10892,7 @@ static void nohz_balancer_kick(struct rq *rq)* around. */ for_each_cpu_and (i,sched_domain_span (sd), nohz.idle_cpus_mask) { -if (sched_asym_prefer (i, cpu)) { +if (sched_asym_prefer (i, cpu, true)) { flags =NOHZ_STATS_KICK |NOHZ_BALANCE_KICK ; goto unlock; }
kernel/sched/sched.h +6 -2
@@ -795,8 +795,12 @@ static inline long se_weight(struct sched_entity *se)return scale_load_down (se->load.weight); } - -static inline bool sched_asym_prefer (int a,int b) +/* +* Used to compare specific CPUs. Also when comparing the preferred CPU of a +* sched group or building the sched domains; in such cases checking the state +* of SMT siblings, if any, is not needed. +*/ +static inline bool sched_asym_prefer (int a,int b,bool check_smt) {return arch_asym_cpu_priority (a) >arch_asym_cpu_priority (b); }
kernel/sched/topology.c +5 -1
@@ -1282,7 +1282,11 @@ static void init_sched_groups_capacity(int cpu, struct sched_domain *sd)for_each_cpu (cpu,sched_group_span (sg)) {if (max_cpu <0 ) max_cpu = cpu; -else if (sched_asym_prefer (cpu, max_cpu)) +/* +* We want the CPU priorities unaffected by the idle +* state of its SMT siblings, if any. +*/ +else if (sched_asym_prefer (cpu, max_cpu, false)) max_cpu = cpu; } sg->asym_prefer_cpu = max_cpu;
Signature · 1 line
2.25.1
PATCH v2 3/7 sched: Teach arch_asym_cpu_priority() the idle state of SMT siblings
In reply to Ricardo NeriSome processors (e.g., Intel processors with ITMT) use asym_packing to balance load between physical cores with SMT. In such case, a core with all its SMT siblings idle is more desirable than another with one or more busy siblings.
Other processors (e.g, Power7 with SMT8) use asym_packing to balance load among SMT siblings of different priority, regardless of their idle state.
Add a new parameter, check_smt, that architectures can use as needed.
---
Changes since v1:
* Introduced this patch.
--- arch/x86/kernel/itmt.c | 2 +- include/linux/sched/topology.h | 2 +- kernel/sched/fair.c | 5 ++++- kernel/sched/sched.h | 3 ++- 4 files changed, 8 insertions(+), 4 deletions(-)
arch/x86/kernel/itmt.c +1 -1
@@ -167,7 +167,7 @@ void sched_clear_itmt_support(void)mutex_unlock (&itmt_update_mutex); } -int arch_asym_cpu_priority (int cpu) +int arch_asym_cpu_priority (int cpu,bool check_smt) {return per_cpu (sched_core_priority, cpu); }
include/linux/sched/topology.h +1 -1
@@ -63,7 +63,7 @@ static inline int cpu_numa_flags(void) }#endif -extern int arch_asym_cpu_priority (int cpu); +extern int arch_asym_cpu_priority (int cpu,bool check_smt);struct sched_domain_attr {int relax_domain_level;
kernel/sched/fair.c +4 -1
@@ -142,8 +142,11 @@ __setup("sched_thermal_decay_shift=", setup_sched_thermal_decay_shift);#ifdef CONFIG_SMP /* * For asym packing, by default the lower numbered CPU has higher priority. +* +* When doing ASYM_PACKING at the "MC" or higher domains, architectures may +* want to check the idle state of the SMT siblngs of @cpu. */ -int __weak arch_asym_cpu_priority (int cpu) +int __weak arch_asym_cpu_priority (int cpu,bool check_smt) {return -cpu; }
kernel/sched/sched.h +2 -1
@@ -802,7 +802,8 @@ static inline long se_weight(struct sched_entity *se)*/ static inline bool sched_asym_prefer (int a,int b,bool check_smt) { -return arch_asym_cpu_priority (a) >arch_asym_cpu_priority (b); +return arch_asym_cpu_priority (a, check_smt) > +arch_asym_cpu_priority (b, check_smt); }struct perf_domain {
Signature · 1 line
2.25.1
Dietmar Eggemann
reviewerRe: [PATCH v2 3/7] sched: Teach arch_asym_cpu_priority() the idle state of SMT siblings
Quoted 9 lines · Ricardo Neri
Some processors (e.g., Intel processors with ITMT) use
asym_packingto balance load between physical cores with SMT. In such case, a core with all its SMT siblings idle is more desirable than another with one or more busy siblings.Other processors (e.g, Power7 with SMT8) use
asym_packingto balance load among SMT siblings of different priority, regardless of their idle state.Add a new parameter,
check_smt, that architectures can use as needed.
[...]
Quoted 2 lines
Changes since v1:
* Introduced this patch.
[...]
kernel/sched/fair.c __setup("sched_thermal_decay_shift=", setup_sched_thermal_decay_shift); · line 146
Quoted 10 lines
kernel/sched/fair.c +2
-0@@ -142,8 +142,11 @@ __setup("sched_thermal_decay_shift=", setup_sched_thermal_decay_shift);#ifdef CONFIG_SMP /* * For asym packing, by default the lower numbered CPU has higher priority. +* +* When doing ASYM_PACKING at the "MC" or higher domains, architectures may
There is this new CLUSTER level between SMT and MC.
kernel/sched/fair.c __setup("sched_thermal_decay_shift=", setup_sched_thermal_decay_shift); · line 147
Quoted 1 line
kernel/sched/fair.c +1
-0+* want to check the idle state of the SMT siblngs of @cpu.
s/siblngs/siblings
The scheduler calls sched_asym_prefer(..., true) from find_busiest_queue(), asym_active_balance() and nohz_balancer_kick() even from SMT layer on !x86. So I guess a bool check_smt wouldn't be sufficient to distinguish whether sched_smt_siblings_idle() should be called or not. To me this comment is a little bit misleading. Not an issue currently since there is only the x86 overwrite right now.
Quoted 6 lines
kernel/sched/fair.c +1
-1*/ -int __weak arch_asym_cpu_priority (int cpu) +int __weak arch_asym_cpu_priority (int cpu,bool check_smt) {return -cpu; }
[...]
Re: [PATCH v2 3/7] sched: Teach arch_asym_cpu_priority() the idle state of SMT siblings
Quoted 18 lines · Dietmar Eggemann
Quoted 9 lines · Ricardo Neri
Some processors (e.g., Intel processors with ITMT) use
asym_packingto balance load between physical cores with SMT. In such case, a core with all its SMT siblings idle is more desirable than another with one or more busy siblings.Other processors (e.g, Power7 with SMT8) use
asym_packingto balance load among SMT siblings of different priority, regardless of their idle state.Add a new parameter,
check_smt, that architectures can use as needed.[...]
Quoted 2 lines
Changes since v1:
* Introduced this patch.[...]
Quoted 12 lines
kernel/sched/fair.c __setup("sched_thermal_decay_shift=", setup_sched_thermal_decay_shift); · line 146
Quoted 10 lines
kernel/sched/fair.c +2
-0@@ -142,8 +142,11 @@ __setup("sched_thermal_decay_shift=", setup_sched_thermal_decay_shift);#ifdef CONFIG_SMP /* * For asym packing, by default the lower numbered CPU has higher priority. +* +* When doing ASYM_PACKING at the "MC" or higher domains, architectures may There is this new CLUSTER level between SMT and MC.
This is a good catch! I will update the comment to say "domains above SMT".
Quoted 7 lines
kernel/sched/fair.c __setup("sched_thermal_decay_shift=", setup_sched_thermal_decay_shift); · line 147
Quoted 2 lines
kernel/sched/fair.c +1
-0+* want to check the idle state of the SMT siblngs of @cpu. s/siblngs/siblings
The scheduler calls
sched_asym_prefer(..., true)fromfind_busiest_queue(),asym_active_balance()andnohz_balancer_kick()
In these places we are comparing two specific CPUs, of which the idle state of its siblings impact their throughput and, in consequence, the decision of attempt to balance load.
In the places were sched_asym_prefer(...., false) is called we compare the destination CPU with a CPU that bears the priority of a sched group, regardless of the idle state of its siblings.
Quoted 1 line
even from SMT layer on !x86.
This is true, but the default arch_asym_cpu_priority ignores check_smt.
Quoted 3 lines
So I guess a
bool check_smtwouldn't be
sufficient to distinguish whethersched_smt_siblings_idle()should be called or not.
But it is the caller who determines whether the idle state of the SMT siblings of @cpu may be relevant.
Quoted 2 lines
To me this comment is a little bit misleading. Not an issue currently since there is only the x86 overwrite right now.
If my justification make sense to you, I can expand the comment to state that the caller decides whether check_smt is needed but arch-specific implementations are free to ignore it.
Thanks and BR,
Ricardo
Dietmar Eggemann
reviewerRe: [PATCH v2 3/7] sched: Teach arch_asym_cpu_priority() the idle state of SMT siblings
Quoted 2 lines · Ricardo Neri
Quoted 1 line · Dietmar Eggemann
On 22/11/2022 21:35, Ricardo Neri wrote:
[...]
Quoted 7 lines
Quoted 6 lines
Quoted 1 line
+ * want to check the idle state of the SMT siblngs of @cpu.
s/siblngs/siblings
The scheduler calls
sched_asym_prefer(..., true)fromfind_busiest_queue(),asym_active_balance()andnohz_balancer_kick()
Quoted 7 lines
In these places we are comparing two specific CPUs, of which the idle state of its siblings impact their throughput and, in consequence, the decision of attempt to balance load.
In the places were
sched_asym_prefer(...., false)is called we compare the destination CPU with a CPU that bears the priority of a sched group, regardless of the idle state of its siblings.
OK.
Quoted 3 lines
Quoted 1 line
even from SMT layer on !x86.
This is true, but the default
arch_asym_cpu_priorityignorescheck_smt.
True.
Quoted 6 lines
Quoted 3 lines
So I guess a
bool check_smtwouldn't be
sufficient to distinguish whethersched_smt_siblings_idle()should be called or not.But it is the caller who determines whether the idle state of the SMT siblings of @cpu may be relevant.
I assume caller being the task scheduler here. Callers with check_smt=true can be called from any SD level with SD_ASYM_PACKING.
Imagine an arch w/ SD_ASYM_PACKING on SMT & MC overwriting arch_asym_cpu_priority(). bool check_smt wouldn't be sufficient to know whether a call to something like sched_smt_siblings_idle() (is_core_idle()) which iterates over cpu_smt_mask(cpu) would make sense.
Quoted 6 lines
Quoted 2 lines
To me this comment is a little bit misleading. Not an issue currently since there is only the x86 overwrite right now.
If my justification make sense to you, I can expand the comment to state that the caller decides whether
check_smtis needed but arch-specific implementations are free to ignore it.
Not a big issue but to me if the task scheduler asks for bool check_smt then archs would have to check to guarantee common behaviour. And the meaning of bool check_smt on SMT is unclear to me. Since only x86 would use this so far it can be adapted later for others if needed.
[...]
Re: [PATCH v2 3/7] sched: Teach arch_asym_cpu_priority() the idle state of SMT siblings
Quoted 29 lines · Dietmar Eggemann
Quoted 2 lines · Ricardo Neri
Quoted 1 line · Dietmar Eggemann
On 22/11/2022 21:35, Ricardo Neri wrote:
[...]
Quoted 7 lines
Quoted 6 lines
Quoted 1 line
+ * want to check the idle state of the SMT siblngs of @cpu.
s/siblngs/siblings
The scheduler calls
sched_asym_prefer(..., true)fromfind_busiest_queue(),asym_active_balance()andnohz_balancer_kick()Quoted 7 lines
In these places we are comparing two specific CPUs, of which the idle state of its siblings impact their throughput and, in consequence, the decision of attempt to balance load.
In the places were
sched_asym_prefer(...., false)is called we compare the destination CPU with a CPU that bears the priority of a sched group, regardless of the idle state of its siblings.OK.
Quoted 3 lines
Quoted 1 line
even from SMT layer on !x86.
This is true, but the default
arch_asym_cpu_priorityignorescheck_smt.True.
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
So I guess a
bool check_smtwouldn't be
sufficient to distinguish whethersched_smt_siblings_idle()should be called or not.But it is the caller who determines whether the idle state of the SMT siblings of @cpu may be relevant.
I assume caller being the task scheduler here.
Yes.
Quoted 2 lines
Callers with
check_smt=truecan be called from any SD level withSD_ASYM_PACKING.
This is true.
Quoted 5 lines
Imagine an arch w/
SD_ASYM_PACKINGon SMT & MC overwritingarch_asym_cpu_priority().bool check_smtwouldn't be sufficient to know whether a call to something likesched_smt_siblings_idle()(is_core_idle()) which iterates overcpu_smt_mask(cpu)would make sense.
Agreed. I was hoping I could get away with this. x86 would not have the the SD_ASYM_PACKING flag at the SMT level and Power7 would ignore check_smt. :)
Callers of sched_asym_prefer() could use the flags of the sched domain to check if we are at the SMT level.
I rescanned the code again and it looks like the needed sched domain flags are available in all the places sched_asym_prefer() is called. The only exception is asym_smt_can_pull_tasks(), but we already know that we don't need such check. (We are looking for the sched group priority, regardless of the idle state of the SMT siblings).
Quoted 8 lines
Quoted 7 lines
Quoted 3 lines
To me this comment is a little bit misleading. Not an issue currently since there is only the x86 overwrite right now.
If my justification make sense to you, I can expand the comment to state that the caller decides whether
check_smtis needed but arch-specific implementations are free to ignore it.
Quoted 5 lines
Not a big issue but to me if the task scheduler asks for
bool check_smtthen archs would have to check to guarantee common behaviour. And the meaning ofbool check_smton SMT is unclear to me. Since only x86 would use this so far it can be adapted later for others if needed.
What is proposed in my previous paragraph should solve this, IMO.
Thanks and BR,
Ricardo
Valentin Schneider
reviewerRe: [PATCH v2 3/7] sched: Teach arch_asym_cpu_priority() the idle state of SMT siblings
Quoted 10 lines · Ricardo Neri
Quoted 4 lines · Dietmar Eggemann
Imagine an arch w/
SD_ASYM_PACKINGon SMT & MC overwritingarch_asym_cpu_priority().bool check_smtwouldn't be sufficient to know whether a call to something likesched_smt_siblings_idle()(is_core_idle()) which iterates overcpu_smt_mask(cpu)would make sense.Agreed. I was hoping I could get away with this. x86 would not have the the
SD_ASYM_PACKINGflag at the SMT level and Power7 would ignorecheck_smt. :)
Quoted 9 lines
Callers of
sched_asym_prefer()could use the flags of the sched domain to check if we are at the SMT level.I rescanned the code again and it looks like the needed sched domain flags are available in all the places
sched_asym_prefer()is called. The only exception isasym_smt_can_pull_tasks(), but we already know that we don't need such check. (We are looking for the sched group priority, regardless of the idle state of the SMT siblings).
Given this is fed back to arch code, another option here would be to feed the topology level this is happening at. You get it via sd->level and it maps back into the arch's sched_domain_topology_level array.
Though the SD flags are probably the more generic solution.
PATCH v2 4/7 sched/fair: Introduce sched_smt_siblings_idle()
In reply to Ricardo NeriArchitectures that implement arch_asym_cpu_priority() may need to know the idle state of the SMT siblings of a CPU. The scheduler has this information and functionality. Expose it.
Move the existing functionality outside of the NUMA code.
---
Changes since v1:
* Introduced this patch.
--- include/linux/sched.h | 2 ++ kernel/sched/fair.c | 39 ++++++++++++++++++++++----------------- 2 files changed, 24 insertions(+), 17 deletions(-)
include/linux/sched.h +2 -0
@@ -2426,4 +2426,6 @@ static inline void sched_core_fork(struct task_struct *p) { }extern void sched_set_stop_task (int cpu,struct task_struct *stop); +extern bool sched_smt_siblings_idle (int cpu); +#endif
kernel/sched/fair.c +22 -17
@@ -1052,6 +1052,28 @@ update_stats_curr_start(struct cfs_rq *cfs_rq, struct sched_entity *se)* Scheduling class queueing methods: */ +static inline bool is_core_idle (int cpu) +{ +#ifdef CONFIG_SCHED_SMT +int sibling; + +for_each_cpu (sibling,cpu_smt_mask (cpu)) { +if (cpu == sibling) +continue ; + +if (!idle_cpu (sibling)) +return false; + } +#endif + +return true; +} + +bool sched_smt_siblings_idle (int cpu) +{ +return is_core_idle (cpu); +} +#ifdef CONFIG_NUMA #define NUMA_IMBALANCE_MIN 2
@@ -1691,23 +1713,6 @@ struct numa_stats {int idle_cpu; }; -static inline bool is_core_idle (int cpu) -{ -#ifdef CONFIG_SCHED_SMT -int sibling; - -for_each_cpu (sibling,cpu_smt_mask (cpu)) { -if (cpu == sibling) -continue ; - -if (!idle_cpu (sibling)) -return false; - } -#endif - -return true; -} -struct task_numa_env {struct task_struct *p;
Signature · 1 line
2.25.1
Dietmar Eggemann
reviewerRe: [PATCH v2 4/7] sched/fair: Introduce sched_smt_siblings_idle()
Quoted 5 lines · Ricardo Neri
Architectures that implement
arch_asym_cpu_priority()may need to know the idle state of the SMT siblings of a CPU. The scheduler has this information and functionality. Expose it.Move the existing functionality outside of the NUMA code.
[...]
Quoted 19 lines
kernel/sched/fair.c +11
-0@@ -1052,6 +1052,28 @@ update_stats_curr_start(struct cfs_rq *cfs_rq, struct sched_entity *se)* Scheduling class queueing methods: */ +static inline bool is_core_idle (int cpu) +{ +#ifdef CONFIG_SCHED_SMT +int sibling; + +for_each_cpu (sibling,cpu_smt_mask (cpu)) { +if (cpu == sibling) +continue ; + +if (!idle_cpu (sibling)) +return false;
kernel/sched/fair.c update_stats_curr_start(struct cfs_rq *cfs_rq, struct sched_entity *se) · line 1075
Quoted 10 lines
kernel/sched/fair.c +10
-0+ } +#endif + +return true; +} + +bool sched_smt_siblings_idle (int cpu) +{ +return is_core_idle (cpu); +}
Nitpick: Can we not just have one exported function for both use-cases:
NUMA and x86 ITMT?
[...]
Re: [PATCH v2 4/7] sched/fair: Introduce sched_smt_siblings_idle()
Quoted 33 lines · Dietmar Eggemann
Quoted 5 lines · Ricardo Neri
Architectures that implement
arch_asym_cpu_priority()may need to know the idle state of the SMT siblings of a CPU. The scheduler has this information and functionality. Expose it.Move the existing functionality outside of the NUMA code.
[...]
Quoted 24 lines
kernel/sched/fair.c +16
-0@@ -1052,6 +1052,28 @@ update_stats_curr_start(struct cfs_rq *cfs_rq, struct sched_entity *se)* Scheduling class queueing methods: */ +static inline bool is_core_idle (int cpu) +{ +#ifdef CONFIG_SCHED_SMT +int sibling; + +for_each_cpu (sibling,cpu_smt_mask (cpu)) { +if (cpu == sibling) +continue ; + +if (!idle_cpu (sibling)) +return false; + } +#endif + +return true; +}
Quoted 8 lines
kernel/sched/fair.c update_stats_curr_start(struct cfs_rq *cfs_rq, struct sched_entity *se) · line 1075
Quoted 5 lines
kernel/sched/fair.c +5
-0+ +bool sched_smt_siblings_idle (int cpu) +{ +return is_core_idle (cpu); +}Nitpick: Can we not just have one exported function for both use-cases:
NUMA and x86 ITMT?
By adding a new function I intend to preserve the inlinig of is_core_idle() in update_numa_stats() (via numa_idle_core(), which is also inline). Do you think there is no value?
A downside of having the new function is that now the code is duplicated in update_numa_stats() and sched_smt_siblings_idle().
I can take your suggestion if losing the inline is OK.
Thanks and BR,
Ricardo
Dietmar Eggemann
reviewerRe: [PATCH v2 4/7] sched/fair: Introduce sched_smt_siblings_idle()
Quoted 2 lines · Ricardo Neri
Quoted 1 line · Dietmar Eggemann
On 22/11/2022 21:35, Ricardo Neri wrote:
[...]
Quoted 11 lines
Quoted 7 lines
kernel/sched/fair.c update_stats_curr_start(struct cfs_rq *cfs_rq, struct sched_entity *se) · line 1075
Quoted 4 lines
kernel/sched/fair.c +4
-0+bool sched_smt_siblings_idle (int cpu) +{ +return is_core_idle (cpu); +}Nitpick: Can we not just have one exported function for both use-cases:
NUMA and x86 ITMT?By adding a new function I intend to preserve the inlinig of
is_core_idle()inupdate_numa_stats()(vianuma_idle_core(), which is also inline). Do you think there is no value?
OK. It's only used in NUMA balancing (task_numa_fault() -> ... -> update_numa_stats()). I can't see that this will have a noticeable perf impact but only benchmark can really tell.
A static inline bool sched_is_core_idle(int cpu) via include/linux/sched/topology.h might work? We have similar functions (like sched_core_cookie_match()` but in the private scheduler header file kernel/sched/sched.h though.
Quoted 4 lines
A downside of having the new function is that now the code is duplicated in
update_numa_stats()andsched_smt_siblings_idle().I can take your suggestion if losing the inline is OK.
I doubt that it will have an impact but can't be sure.
[...]
Valentin Schneider
reviewerRe: [PATCH v2 4/7] sched/fair: Introduce sched_smt_siblings_idle()
In reply to Ricardo NeriQuoted 6 lines · Ricardo Neri
Architectures that implement
arch_asym_cpu_priority()may need to know the idle state of the SMT siblings of a CPU. The scheduler has this information and functionality. Expose it.Move the existing functionality outside of the NUMA code.
test_idle_cores() does something similar without an iteration, did you consider using that instead?
Quoted 91 lines
Cc Ben Segall <[email protected]>Cc Daniel Bristot de Oliveira <[email protected]>Cc Dietmar Eggemann <[email protected]>Cc Len Brown <[email protected]>Cc Mel Gorman <[email protected]>Cc Rafael J. Wysocki <[email protected]>Cc Srinivas Pandruvada <[email protected]>Cc Steven Rostedt <[email protected]>Cc Tim C. Chen <[email protected]>Cc Valentin Schneider <[email protected]>Signed-off-by Ricardo Neri <[email protected]>---
Changes since v1:
* Introduced this patch.--- include/linux/sched.h | 2 ++ kernel/sched/fair.c | 39 ++++++++++++++++++++++----------------- 2 files changed, 24 insertions(+), 17 deletions(-)include/linux/sched.h +2
-0@@ -2426,4 +2426,6 @@ static inline void sched_core_fork(struct task_struct *p) { }extern void sched_set_stop_task (int cpu,struct task_struct *stop); +extern bool sched_smt_siblings_idle (int cpu); +#endif kernel/sched/fair.c +22
-17@@ -1052,6 +1052,28 @@ update_stats_curr_start(struct cfs_rq *cfs_rq, struct sched_entity *se)* Scheduling class queueing methods: */ +static inline bool is_core_idle (int cpu) +{ +#ifdef CONFIG_SCHED_SMT +int sibling; + +for_each_cpu (sibling,cpu_smt_mask (cpu)) { +if (cpu == sibling) +continue ; + +if (!idle_cpu (sibling)) +return false; + } +#endif + +return true; +} + +bool sched_smt_siblings_idle (int cpu) +{ +return is_core_idle (cpu); +} +#ifdef CONFIG_NUMA #define NUMA_IMBALANCE_MIN 2@@ -1691,23 +1713,6 @@ struct numa_stats {int idle_cpu; }; -static inline bool is_core_idle (int cpu) -{ -#ifdef CONFIG_SCHED_SMT -int sibling; - -for_each_cpu (sibling,cpu_smt_mask (cpu)) { -if (cpu == sibling) -continue ; - -if (!idle_cpu (sibling)) -return false; - } -#endif - -return true; -} -struct task_numa_env {struct task_struct *p;Signature · 1 line
2.25.1
Re: [PATCH v2 4/7] sched/fair: Introduce sched_smt_siblings_idle()
Quoted 10 lines · Valentin Schneider
Quoted 6 lines · Ricardo Neri
Architectures that implement
arch_asym_cpu_priority()may need to know the idle state of the SMT siblings of a CPU. The scheduler has this information and functionality. Expose it.Move the existing functionality outside of the NUMA code.
test_idle_cores()does something similar without an iteration, did you consider using that instead?
IIUC, test_idle_cores() returns true if there is at least one idle core in the package. In my case, I need to know the idle state of only the SMT siblings of a specific CPU. Am I missing something?
Thanks and BR,
Ricardo
Re: [PATCH v2 4/7] sched/fair: Introduce sched_smt_siblings_idle()
Quoted 9 lines · Ricardo Neri
Quoted 8 lines · Valentin Schneider
Quoted 6 lines · Ricardo Neri
Architectures that implement
arch_asym_cpu_priority()may need to know the idle state of the SMT siblings of a CPU. The scheduler has this information and functionality. Expose it.Move the existing functionality outside of the NUMA code.
Quoted 7 lines
Quoted 2 lines
test_idle_cores()does something similar without an iteration, did you consider using that instead?IIUC,
test_idle_cores()returns true if there is at least one idle core in the package. In my case, I need to know the idle state of only the SMT siblings of a specific CPU. Am I missing something?
I guess a similar one is select_idle_core(), but it also consider the CPU with SCHED_IDLE task as idle. Is CPU with SCHED_IDLE task a candidate in your scenario?
thanks,
Chenyu
Quoted 2 lines
Thanks and BR,
Ricardo
Re: [PATCH v2 4/7] sched/fair: Introduce sched_smt_siblings_idle()
Quoted 13 lines · Chen Yu
Quoted 12 lines · Ricardo Neri
Quoted 10 lines · Valentin Schneider
Quoted 6 lines · Ricardo Neri
Architectures that implement
arch_asym_cpu_priority()may need to know the idle state of the SMT siblings of a CPU. The scheduler has this information and functionality. Expose it.Move the existing functionality outside of the NUMA code.
test_idle_cores()does something similar without an iteration, did you consider using that instead?
Quoted 7 lines
Quoted 4 lines
IIUC,
test_idle_cores()returns true if there is at least one idle core in the package. In my case, I need to know the idle state of only the SMT siblings of a specific CPU. Am I missing something?I guess a similar one is
select_idle_core(), but it also consider the CPU withSCHED_IDLEtask as idle. Is CPU withSCHED_IDLEtask a candidate in your scenario?
However, we are not looking for an idle CPU. We want to know the idle state of the siblings of a CPU. I see that select_idle_core() uses available_idle_cpu(), which in turn uses idle_cpu(). is_core_idle() also uses it.
As per 943d355d7fee ("sched/core: Distinguish between idle_cpu() calls based on desired effect, introduce available_idle_cpu()") the load balancer can just call idle_cpu().
Thanks and BR,
Ricardo
Valentin Schneider
reviewerRe: [PATCH v2 4/7] sched/fair: Introduce sched_smt_siblings_idle()
In reply to Ricardo NeriQuoted 9 lines · Ricardo Neri
Quoted 8 lines · Valentin Schneider
Quoted 6 lines · Ricardo Neri
Architectures that implement
arch_asym_cpu_priority()may need to know the idle state of the SMT siblings of a CPU. The scheduler has this information and functionality. Expose it.Move the existing functionality outside of the NUMA code.
Quoted 7 lines
Quoted 2 lines
test_idle_cores()does something similar without an iteration, did you consider using that instead?IIUC,
test_idle_cores()returns true if there is at least one idle core in the package. In my case, I need to know the idle state of only the SMT siblings of a specific CPU. Am I missing something?
No, you're right, clearly I needed that end of the year break - sorry for the noise.
Quoted 2 lines
Thanks and BR,
Ricardo
PATCH v2 5/7 x86/sched: Remove SD_ASYM_PACKING from the "SMT" domain
In reply to Ricardo NeriThere is no difference between any of the SMT siblings of a physical core. asym_packing load balancing is not needed among siblings.
When balancing load among physical cores, the scheduler now considers the state of the siblings when checking the priority of a CPU.
---
Changes since v1:
* Introduced this patch.
--- arch/x86/kernel/smpboot.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/kernel/smpboot.c +1 -1
@@ -545,7 +545,7 @@ static int x86_core_flags(void)#ifdef CONFIG_SCHED_SMT static int x86_smt_flags (void ) { -return cpu_smt_flags () |x86_sched_itmt_flags (); +return cpu_smt_flags (); }#endif #ifdef CONFIG_SCHED_CLUSTER
Signature · 1 line
2.25.1
Re: [PATCH v2 5/7] x86/sched: Remove SD_ASYM_PACKING from the "SMT" domain
Hi Ricardo,
Quoted 31 lines · Ricardo Neri
There is no difference between any of the SMT siblings of a physical core.
asym_packingload balancing is not needed among siblings.When balancing load among physical cores, the scheduler now considers the state of the siblings when checking the priority of a CPU.
Cc Ben Segall <[email protected]>Cc Daniel Bristot de Oliveira <[email protected]>Cc Dietmar Eggemann <[email protected]>Cc Len Brown <[email protected]>Cc Mel Gorman <[email protected]>Cc Rafael J. Wysocki <[email protected]>Cc Srinivas Pandruvada <[email protected]>Cc Steven Rostedt <[email protected]>Cc Tim C. Chen <[email protected]>Cc Valentin Schneider <[email protected]>Signed-off-by Ricardo Neri <[email protected]>---
Changes since v1:
* Introduced this patch.--- arch/x86/kernel/smpboot.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)arch/x86/kernel/smpboot.c +0
-0
arch/x86/kernel/smpboot.c static int x86_core_flags(void) · line 548
Quoted 5 lines
arch/x86/kernel/smpboot.c +1
-1@@ -545,7 +545,7 @@ static int x86_core_flags(void)#ifdef CONFIG_SCHED_SMT static int x86_smt_flags (void ) { -return cpu_smt_flags () |x86_sched_itmt_flags (); +return cpu_smt_flags ();
Based on:
kernel/sched/topology.c:
sd = highest_flag_domain(cpu, SD_ASYM_PACKING); rcu_assign_pointer(per_cpu(sd_asym_packing, cpu), sd);
and described at:
include/linux/sched/sd_flags.h:
/*
* Place busy tasks earlier in the domain
*
* SHARED_CHILD: Usually set on the SMT level. Technically could be set further
* up, but currently assumed to be set from the base domain
* upwards (see update_top_cache_domain()).
* NEEDS_GROUPS: Load balancing flag.
*/SD_FLAG(SD_ASYM_PACKING, SDF_SHARED_CHILD | SDF_NEEDS_GROUPS)
doesn't your change result in sd_asym_packing being NULL?
The SD_ASYM_PACKING flag requires all children of a domain to have it set as well. So having SMT not setting the flag, while CLUSTER and MC having set the flag would result in a broken topology, right?
Thanks,
Ionela.
Quoted 7 lines
arch/x86/kernel/smpboot.c +0
-0}#endif #ifdef CONFIG_SCHED_CLUSTER Signature · 1 line
2.25.1
Re: [PATCH v2 5/7] x86/sched: Remove SD_ASYM_PACKING from the "SMT" domain
Quoted 1 line · Ionela Voinescu
Hi Ricardo,
Hi Ionela,
Thank you very much for your feedback!
Quoted 58 lines · Ricardo Neri
Quoted 31 lines
There is no difference between any of the SMT siblings of a physical core.
asym_packingload balancing is not needed among siblings.When balancing load among physical cores, the scheduler now considers the state of the siblings when checking the priority of a CPU.
Cc Ben Segall <[email protected]>Cc Daniel Bristot de Oliveira <[email protected]>Cc Dietmar Eggemann <[email protected]>Cc Len Brown <[email protected]>Cc Mel Gorman <[email protected]>Cc Rafael J. Wysocki <[email protected]>Cc Srinivas Pandruvada <[email protected]>Cc Steven Rostedt <[email protected]>Cc Tim C. Chen <[email protected]>Cc Valentin Schneider <[email protected]>Signed-off-by Ricardo Neri <[email protected]>---
Changes since v1:
* Introduced this patch.--- arch/x86/kernel/smpboot.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)arch/x86/kernel/smpboot.c +0
-0arch/x86/kernel/smpboot.c static int x86_core_flags(void) · line 548
Quoted 5 lines
arch/x86/kernel/smpboot.c +1
-1@@ -545,7 +545,7 @@ static int x86_core_flags(void)#ifdef CONFIG_SCHED_SMT static int x86_smt_flags (void ) { -return cpu_smt_flags () |x86_sched_itmt_flags (); +return cpu_smt_flags ();Based on:
kernel/sched/topology.c:
sd =highest_flag_domain(cpu, SD_ASYM_PACKING);rcu_assign_pointer(per_cpu(sd_asym_packing, cpu), sd);and described at:
include/linux/sched/sd_flags.h:
/*
* Place busy tasks earlier in the domain
*
*SHARED_CHILD: Usually set on the SMT level. Technically could be set further
* up, but currently assumed to be set from the base domain
* upwards (seeupdate_top_cache_domain()).
*NEEDS_GROUPS: Load balancing flag.
*/SD_FLAG(SD_ASYM_PACKING, SDF_SHARED_CHILD | SDF_NEEDS_GROUPS)
Quoted 1 line
doesn't your change result in
sd_asym_packingbeing NULL?
Yes. This is a good catch. Thanks!
Quoted 4 lines
The
SD_ASYM_PACKINGflag requires all children of a domain to have it set as well. So having SMT not setting the flag, while CLUSTER and MC having set the flag would result in a broken topology, right?
I'd say that highest_flag_domain(..., flag) requires all children to have flag, but clearly the comment you quote allows for SD_ASYM_PACKING to be located in upper domains.
Perhaps this can be fixed with a variant of highest_flag_domain() that do not require all children to have the flag?
Thanks and BR,
Ricardo
Valentin Schneider
reviewerRe: [PATCH v2 5/7] x86/sched: Remove SD_ASYM_PACKING from the "SMT" domain
Quoted 20 lines · Ricardo Neri
Quoted 19 lines · Ionela Voinescu
Based on:
kernel/sched/topology.c:
sd =highest_flag_domain(cpu, SD_ASYM_PACKING);rcu_assign_pointer(per_cpu(sd_asym_packing, cpu), sd);and described at:
include/linux/sched/sd_flags.h:
/*
* Place busy tasks earlier in the domain
*
*SHARED_CHILD: Usually set on the SMT level. Technically could be set further
* up, but currently assumed to be set from the base domain
* upwards (seeupdate_top_cache_domain()).
*NEEDS_GROUPS: Load balancing flag.
*/SD_FLAG(SD_ASYM_PACKING, SDF_SHARED_CHILD | SDF_NEEDS_GROUPS)
Quoted 4 lines
Quoted 1 line
doesn't your change result in
sd_asym_packingbeing NULL?Yes. This is a good catch. Thanks!
Nice to see those being useful :-) FYI if you run your kernel with CONFIG_SCHED_DEBUG=y and sched_debug on the cmdline, you should get a warning at boot time from the topology debug code checking assertions against those flags.
Quoted 5 lines
Quoted 4 lines
The
SD_ASYM_PACKINGflag requires all children of a domain to have it set as well. So having SMT not setting the flag, while CLUSTER and MC having set the flag would result in a broken topology, right?
Quoted 7 lines
I'd say that
highest_flag_domain(..., flag)requires all children to haveflag, but clearly the comment you quote allows forSD_ASYM_PACKINGto be located in upper domains.Perhaps this can be fixed with a variant of
highest_flag_domain()that do not require all children to have the flag?
So I gave that flag SDF_SHARED_CHILD because its cached SD pointer was set up using highest_flag_domain(). Looking for the highest level where it is set matches how it is used in nohz_balancer_kick(), so you might want a new helper.
With that said, so far all but one flag (SD_PREFER_SIBLING, and that's because of big.LITTLE woes) follow the SDF_SHARED_{CHILD, PARENT} pattern, if SD_ASYM_PACKING no longer does then we need to think whether we're trying to make it do funky things. I need to look at the rest of your series to get an idea, that unfortunately won't be today but it's now in my todolist.
Quoted 2 lines
Thanks and BR,
Ricardo
Re: [PATCH v2 5/7] x86/sched: Remove SD_ASYM_PACKING from the "SMT" domain
Quoted 23 lines · Valentin Schneider
Quoted 22 lines · Ricardo Neri
Quoted 20 lines · Ionela Voinescu
Based on:
kernel/sched/topology.c:
sd =highest_flag_domain(cpu, SD_ASYM_PACKING);rcu_assign_pointer(per_cpu(sd_asym_packing, cpu), sd);and described at:
include/linux/sched/sd_flags.h:
/*
* Place busy tasks earlier in the domain
*
*SHARED_CHILD: Usually set on the SMT level. Technically could be set further
* up, but currently assumed to be set from the base domain
* upwards (seeupdate_top_cache_domain()).
*NEEDS_GROUPS: Load balancing flag.
*/SD_FLAG(SD_ASYM_PACKING, SDF_SHARED_CHILD | SDF_NEEDS_GROUPS)doesn't your change result in
sd_asym_packingbeing NULL?
Quoted 7 lines
Quoted 2 lines
Yes. This is a good catch. Thanks!
Nice to see those being useful :-) FYI if you run your kernel with
CONFIG_SCHED_DEBUG=yandsched_debugon the cmdline, you should get a warning at boot time from the topology debug code checking assertions against those flags.
Thanks! I missed this warning. Indeed, it is there.
Quoted 10 lines
Quoted 10 lines
Quoted 5 lines
The
SD_ASYM_PACKINGflag requires all children of a domain to have it set as well. So having SMT not setting the flag, while CLUSTER and MC having set the flag would result in a broken topology, right?I'd say that
highest_flag_domain(..., flag)requires all children to haveflag, but clearly the comment you quote allows forSD_ASYM_PACKINGto be located in upper domains.
Quoted 8 lines
Quoted 3 lines
Perhaps this can be fixed with a variant of
highest_flag_domain()that do not require all children to have the flag?So I gave that flag
SDF_SHARED_CHILDbecause its cached SD pointer was set up usinghighest_flag_domain(). Looking for the highest level where it is set matches how it is used innohz_balancer_kick(), so you might want a new helper.
Right. I was thinking on a highest_flag_domain_weak() or a changing highest_flag_domain(..., bool shared_child).
Quoted 5 lines
With that said, so far all but one flag (
SD_PREFER_SIBLING, and that's because of big.LITTLE woes) follow the SDF_SHARED_{CHILD, PARENT} pattern, ifSD_ASYM_PACKINGno longer does then we need to think whether we're trying to make it do funky things.
My thesis is that x86 does not need the SD_ASYM_PACKING flag at the SMT level because all SMT siblings are identical. There are cores of higher priority at the "MC" level (maybe in the future at the "CLS" level).
Power7 is fine because it only uses SD_ASYM_PACKING at the SMT level.
Quoted 3 lines
I need to look at the rest of your
series to get an idea, that unfortunately won't be today but it's now in my todolist.
Thank you!
BR,
Ricardo
Valentin Schneider
reviewerRe: [PATCH v2 5/7] x86/sched: Remove SD_ASYM_PACKING from the "SMT" domain
Quoted 6 lines · Ricardo Neri
Quoted 4 lines · Valentin Schneider
With that said, so far all but one flag (
SD_PREFER_SIBLING, and that's because of big.LITTLE woes) follow the SDF_SHARED_{CHILD, PARENT} pattern, ifSD_ASYM_PACKINGno longer does then we need to think whether we're trying to make it do funky things.
Quoted 6 lines
My thesis is that x86 does not need the
SD_ASYM_PACKINGflag at the SMT level because all SMT siblings are identical. There are cores of higher priority at the "MC" level (maybe in the future at the "CLS" level).Power7 is fine because it only uses
SD_ASYM_PACKINGat the SMT level.
So with what I groked from your series, I agree with you, x86 shouldn't need it at SMT level.
What about the below?
---
include/linux/sched/sd_flags.h +4 -4
@@ -132,12 +132,12 @@ SD_FLAG(SD_SERIALIZE, SDF_SHARED_PARENT | SDF_NEEDS_GROUPS)/* * Place busy tasks earlier in the domain * -* SHARED_CHILD: Usually set on the SMT level. Technically could be set further -* up, but currently assumed to be set from the base domain -* upwards (see update_top_cache_domain()). +* SHARED_PARENT: Usually set on the SMT level. Can be set further up if all +* siblings of an SMT core are identical, but SMT cores themselves +* have different priorites. * NEEDS_GROUPS: Load balancing flag. */ -SD_FLAG (SD_ASYM_PACKING ,SDF_SHARED_CHILD |SDF_NEEDS_GROUPS ) +SD_FLAG (SD_ASYM_PACKING ,SDF_SHARED_PARENT |SDF_NEEDS_GROUPS )/* * Prefer to place tasks in a sibling domain
kernel/sched/sched.h +24 -9
@@ -1772,6 +1772,19 @@ queue_balance_callback(struct rq *rq,for (__sd =rcu_dereference_check_sched_domain (cpu_rq (cpu)->sd); \ __sd; __sd = __sd->parent) +static inline struct sched_domain *__highest_flag_domain(struct sched_domain *sd, int flag) +{ +struct sched_domain *hsd =NULL ; + +for (; sd; sd = sd->parent) { +if (!(sd->flags & flag)) +break ; + hsd = sd; + } + +return hsd; +} +/** * highest_flag_domain - Return highest sched_domain containing flag. * @cpu: The CPU whose highest level of sched domain is to
@@ -1783,15 +1796,7 @@ queue_balance_callback(struct rq *rq,*/ static inline struct sched_domain *highest_flag_domain (int cpu,int flag) { -struct sched_domain *sd, *hsd =NULL ; - -for_each_domain (cpu, sd) { -if (!(sd->flags & flag)) -break ; - hsd = sd; - } - -return hsd; +return __highest_flag_domain (rcu_dereference_check_sched_domain (cpu_rq (cpu)->sd), flag); }static inline struct sched_domain *lowest_flag_domain (int cpu,int flag)
@@ -1806,6 +1811,16 @@ static inline struct sched_domain *lowest_flag_domain(int cpu, int flag)return sd; } +static inline struct sched_domain *highest_parent_flag_domain (int cpu,int flag) +{ +struct sched_domain *sd; + +SCHED_WARN_ON (!(sd_flag_debug[ilog2 (flag)].meta_flags &SDF_SHARED_PARENT )); + + sd =lowest_flag_domain (cpu, flag); +return __highest_flag_domain (sd, flag); +} + DECLARE_PER_CPU(struct sched_domain __rcu *, sd_llc);DECLARE_PER_CPU (int, sd_llc_size);DECLARE_PER_CPU (int, sd_llc_id);
kernel/sched/topology.c +1 -1
@@ -692,7 +692,7 @@ static void update_top_cache_domain(int cpu) sd =lowest_flag_domain (cpu,SD_NUMA );rcu_assign_pointer (per_cpu (sd_numa, cpu), sd); - sd =highest_flag_domain (cpu,SD_ASYM_PACKING ); + sd =highest_parent_flag_domain (cpu,SD_ASYM_PACKING );rcu_assign_pointer (per_cpu (sd_asym_packing, cpu), sd); sd =lowest_flag_domain (cpu,SD_ASYM_CPUCAPACITY_FULL );
Re: [PATCH v2 5/7] x86/sched: Remove SD_ASYM_PACKING from the "SMT" domain
Quoted 29 lines · Valentin Schneider
Quoted 6 lines · Ricardo Neri
Quoted 4 lines · Valentin Schneider
With that said, so far all but one flag (
SD_PREFER_SIBLING, and that's because of big.LITTLE woes) follow the SDF_SHARED_{CHILD, PARENT} pattern, ifSD_ASYM_PACKINGno longer does then we need to think whether we're trying to make it do funky things.Quoted 6 lines
My thesis is that x86 does not need the
SD_ASYM_PACKINGflag at the SMT level because all SMT siblings are identical. There are cores of higher priority at the "MC" level (maybe in the future at the "CLS" level).Power7 is fine because it only uses
SD_ASYM_PACKINGat the SMT level.So with what I groked from your series, I agree with you, x86 shouldn't need it at SMT level.
What about the below?
---
include/linux/sched/sd_flags.h +0
-0@@ -132,12 +132,12 @@ SD_FLAG(SD_SERIALIZE, SDF_SHARED_PARENT | SDF_NEEDS_GROUPS)/* * Place busy tasks earlier in the domain *
include/linux/sched/sd_flags.h SD_FLAG(SD_SERIALIZE, SDF_SHARED_PARENT | SDF_NEEDS_GROUPS) · line 140
Quoted 10 lines
include/linux/sched/sd_flags.h +4
-4-* SHARED_CHILD: Usually set on the SMT level. Technically could be set further -* up, but currently assumed to be set from the base domain -* upwards (see update_top_cache_domain()). +* SHARED_PARENT: Usually set on the SMT level. Can be set further up if all +* siblings of an SMT core are identical, but SMT cores themselves +* have different priorites. * NEEDS_GROUPS: Load balancing flag. */ -SD_FLAG (SD_ASYM_PACKING ,SDF_SHARED_CHILD |SDF_NEEDS_GROUPS ) +SD_FLAG (SD_ASYM_PACKING ,SDF_SHARED_PARENT |SDF_NEEDS_GROUPS )
But this would not work for Power7. It only has SD_ASYM_PACKING in the SMT sched domain. Must it have either of these flags?
In Power7 SMT siblings have the different priority but, IIUC, physical cores are identical.
It seems to me that asym_packing is specific to a domain.
Thanks and BR,
Ricardo
Valentin Schneider
reviewerRe: [PATCH v2 5/7] x86/sched: Remove SD_ASYM_PACKING from the "SMT" domain
Quoted 14 lines · Ricardo Neri
Quoted 13 lines · Valentin Schneider
include/linux/sched/sd_flags.h +2
-3@@ -132,12 +132,12 @@ SD_FLAG(SD_SERIALIZE, SDF_SHARED_PARENT | SDF_NEEDS_GROUPS)/* * Place busy tasks earlier in the domain * -* SHARED_CHILD: Usually set on the SMT level. Technically could be set further -* up, but currently assumed to be set from the base domain -* upwards (see update_top_cache_domain()). +* SHARED_PARENT: Usually set on the SMT level. Can be set further up if all +* siblings of an SMT core are identical, but SMT cores themselves
Quoted 9 lines
include/linux/sched/sd_flags.h SD_FLAG(SD_SERIALIZE, SDF_SHARED_PARENT | SDF_NEEDS_GROUPS) · line 140
Quoted 5 lines
include/linux/sched/sd_flags.h +2
-1+* have different priorites. * NEEDS_GROUPS: Load balancing flag. */ -SD_FLAG (SD_ASYM_PACKING ,SDF_SHARED_CHILD |SDF_NEEDS_GROUPS ) +SD_FLAG (SD_ASYM_PACKING ,SDF_SHARED_PARENT |SDF_NEEDS_GROUPS )But this would not work for Power7. It only has
SD_ASYM_PACKINGin the SMT sched domain. Must it have either of these flags?
It's not mandatory, but making sure SD flags conform to either of them means the topology debugging infra can help spot misshapen topologies...
Quoted 3 lines
In Power7 SMT siblings have the different priority but, IIUC, physical cores are identical.
...But you're right, this doesn't work with Power7 as it would need SD_ASYM_PACKING all the way up the topology to conform with SDF_SHARED_PARENT, which clearly doesn't work with how Power7 uses asym_packing.
Quoted 2 lines
It seems to me that
asym_packingis specific to a domain.
For Power7 it is, since the asymmetry is only between siblings of a given core. For other systems where the asymmetry is between cores, that could theoretically affect several levels. Consider:
DIE [ ] MC [ ][ ] SMT [ ][ ][ ][ ] CPU 0 1 2 3 4 5 6 7 prio 3 3 2 2 1 1 0 0
As done in your patch, here asym_packing doesn't make sense for SMT, but it does for MC and DIE.
Anywho, I think what this means if we should drop the SDF_SHARED_* metaflag for SD_ASYM_PACKING, unless we can think of a nice way to programmatically describe how SD_ASYM_PACKING should be set.
Quoted 2 lines
Thanks and BR,
Ricardo
Re: [PATCH v2 5/7] x86/sched: Remove SD_ASYM_PACKING from the "SMT" domain
Quoted 51 lines · Valentin Schneider
Quoted 14 lines · Ricardo Neri
Quoted 13 lines · Valentin Schneider
include/linux/sched/sd_flags.h +2
-3@@ -132,12 +132,12 @@ SD_FLAG(SD_SERIALIZE, SDF_SHARED_PARENT | SDF_NEEDS_GROUPS)/* * Place busy tasks earlier in the domain * -* SHARED_CHILD: Usually set on the SMT level. Technically could be set further -* up, but currently assumed to be set from the base domain -* upwards (see update_top_cache_domain()). +* SHARED_PARENT: Usually set on the SMT level. Can be set further up if all +* siblings of an SMT core are identical, but SMT cores themselves Quoted 9 lines
include/linux/sched/sd_flags.h SD_FLAG(SD_SERIALIZE, SDF_SHARED_PARENT | SDF_NEEDS_GROUPS) · line 140
Quoted 5 lines
include/linux/sched/sd_flags.h +2
-1+* have different priorites. * NEEDS_GROUPS: Load balancing flag. */ -SD_FLAG (SD_ASYM_PACKING ,SDF_SHARED_CHILD |SDF_NEEDS_GROUPS ) +SD_FLAG (SD_ASYM_PACKING ,SDF_SHARED_PARENT |SDF_NEEDS_GROUPS )But this would not work for Power7. It only has
SD_ASYM_PACKINGin the SMT sched domain. Must it have either of these flags?It's not mandatory, but making sure SD flags conform to either of them means the topology debugging infra can help spot misshapen topologies...
Quoted 3 lines
In Power7 SMT siblings have the different priority but, IIUC, physical cores are identical.
...But you're right, this doesn't work with Power7 as it would need
SD_ASYM_PACKINGall the way up the topology to conform withSDF_SHARED_PARENT, which clearly doesn't work with how Power7 usesasym_packing.Quoted 2 lines
It seems to me that
asym_packingis specific to a domain.For Power7 it is, since the asymmetry is only between siblings of a given core. For other systems where the asymmetry is between cores, that could theoretically affect several levels. Consider:
DIE [ ] MC [ ][ ] SMT [ ][ ][ ][ ] CPU 0 1 2 3 4 5 6 7 prio 3 3 2 2 1 1 0 0
Quoted 6 lines
As done in your patch, here
asym_packingdoesn't make sense for SMT, but it does for MC and DIE.Anywho, I think what this means if we should drop the SDF_SHARED_* metaflag for
SD_ASYM_PACKING, unless we can think of a nice way to programmatically describe howSD_ASYM_PACKINGshould be set.
Perhaps it can be done by adding an extra check for sg::asym_prefer_cpu. In the example you give, DIE does not need SD_ASYM_PACKING if asym_prefer_cpu all the MC sched groups of have the same priority. This would satisfy both Power7 and x86.
This assumes that priorities are available when checking the sanity of the topology.
Thanks and BR,
Ricardo
PATCH v2 6/7 x86/sched/itmt: Give all SMT siblings of a core the same priority
In reply to Ricardo NeriCurrently, each SMT sibling is given a priority that ensures that in partially busy systems load is evenly spread among cores.
There is, however, no difference in performance among the SMT siblings of a core.
Having different priorities for each SMT sibling triggers unnecessary load balancing towards the higher-priority sibling.
---
Changes since v1:
* Reworded commit message for clarity.
--- arch/x86/kernel/itmt.c | 23 +++++------------------ 1 file changed, 5 insertions(+), 18 deletions(-)
arch/x86/kernel/itmt.c +5 -18
@@ -174,32 +174,19 @@ int arch_asym_cpu_priority(int cpu, bool check_smt)/** * sched_set_itmt_core_prio() - Set CPU priority based on ITMT -* @prio: Priority of cpu core -* @core_cpu: The cpu number associated with the core +* @prio: Priority of @cpu +* @cpu: The CPU number * * The pstate driver will find out the max boost frequency * and call this function to set a priority proportional -* to the max boost frequency. CPU with higher boost +* to the max boost frequency. CPUs with higher boost * frequency will receive higher priority. * * No need to rebuild sched domain after updating * the CPU priorities. The sched domains have no * dependency on CPU priorities. */ -void sched_set_itmt_core_prio (int prio,int core_cpu) +void sched_set_itmt_core_prio (int prio,int cpu) { -int cpu, i =1 ; - -for_each_cpu (cpu,topology_sibling_cpumask (core_cpu)) { -int smt_prio; - -/* -* Ensure that the siblings are moved to the end -* of the priority chain and only used when -* all other high priority cpus are out of capacity. -*/ - smt_prio = prio * smp_num_siblings / (i * i); -per_cpu (sched_core_priority, cpu) = smt_prio; - i++; - } +per_cpu (sched_core_priority, cpu) = prio; }
Signature · 1 line
2.25.1
PATCH v2 7/7 x86/sched/itmt: Consider the idle state of SMT siblings
In reply to Ricardo NeriCores with more than one busy SMT sibling need help from fully idle cores, even if they have lower priority. Return a priority of 0 if any of the SMT siblings of a CPU are busy. This lets lower-priority idle cores pull tasks from the CPU.
Only do this if the scheduler cares about the idle state of the siblings.
---
Changes since v1:
* Introduced this patch.
--- arch/x86/kernel/itmt.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
arch/x86/kernel/itmt.c +4 -1
@@ -169,7 +169,10 @@ void sched_clear_itmt_support(void)int arch_asym_cpu_priority (int cpu,bool check_smt) { -return per_cpu (sched_core_priority, cpu); +if (!check_smt ||sched_smt_siblings_idle (cpu)) +return per_cpu (sched_core_priority, cpu); + +return 0 ; }/**
Signature · 1 line
2.25.1