sched/fair: Fix inaccurate tally of ttwu_move_affine
PATCH 1/1 sched/fair: Fix inaccurate tally of ttwu_move_affine
There are scenarios where non-affine wakeups are incorrectly counted as affine wakeups by schedstats.
When wake_affine_idle() returns prev_cpu which doesn't equal to nr_cpumask_bits, it will slip through the check: target == nr_cpumask_bits in wake_affine() and be counted as if target == this_cpu in schedstats.
Replace target == nr_cpumask_bits with target != this_cpu to make sure affine wakeups are accurately tallied.
--- kernel/sched/fair.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/sched/fair.c +1 -1
@@ -6114,7 +6114,7 @@ static int wake_affine(struct sched_domain *sd, struct task_struct *p, target =wake_affine_weight (sd, p, this_cpu, prev_cpu, sync);schedstat_inc (p->stats.nr_wakeups_affine_attempts); -if (target == nr_cpumask_bits) +if (target != this_cpu)return prev_cpu;schedstat_inc (sd->ttwu_move_affine);
Signature · 1 line
2.31.1
Peter Zijlstra
maintainerQuoted 22 lines · Libo Chen
There are scenarios where non-affine wakeups are incorrectly counted as affine wakeups by schedstats.
When
wake_affine_idle()returnsprev_cpuwhich doesn't equal tonr_cpumask_bits, it will slip through the check: target ==nr_cpumask_bitsinwake_affine()and be counted as if target ==this_cpuin schedstats.Replace target ==
nr_cpumask_bitswith target !=this_cputo make sure affine wakeups are accurately tallied.Fixes 806486c377e33 (sched/fair: Do not migrate if the prev_cpu is idle)Suggested-by Daniel Jordan <[email protected]>Signed-off-by Libo Chen <[email protected]>--- kernel/sched/fair.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)kernel/sched/fair.c +0
-0
kernel/sched/fair.c static int wake_affine(struct sched_domain *sd, struct task_struct *p, · line 6120
Quoted 8 lines
kernel/sched/fair.c +1
-1@@ -6114,7 +6114,7 @@ static int wake_affine(struct sched_domain *sd, struct task_struct *p, target =wake_affine_weight (sd, p, this_cpu, prev_cpu, sync);schedstat_inc (p->stats.nr_wakeups_affine_attempts); -if (target == nr_cpumask_bits) +if (target != this_cpu)return prev_cpu;schedstat_inc (sd->ttwu_move_affine);
This not only changes the accounting but also the placement, no?
Quoted 25 lines · Peter Zijlstra
Quoted 24 lines · Libo Chen
There are scenarios where non-affine wakeups are incorrectly counted as affine wakeups by schedstats.
When
wake_affine_idle()returnsprev_cpuwhich doesn't equal tonr_cpumask_bits, it will slip through the check: target ==nr_cpumask_bitsinwake_affine()and be counted as if target ==this_cpuin schedstats.Replace target ==
nr_cpumask_bitswith target !=this_cputo make sure affine wakeups are accurately tallied.Fixes 806486c377e33 (sched/fair: Do not migrate if the prev_cpu is idle)Suggested-by Daniel Jordan <[email protected]>Signed-off-by Libo Chen <[email protected]>--- kernel/sched/fair.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)kernel/sched/fair.c +0
-0@@ -6114,7 +6114,7 @@ static int wake_affine(struct sched_domain *sd, struct task_struct *p, target =wake_affine_weight (sd, p, this_cpu, prev_cpu, sync);
Quoted 7 lines
kernel/sched/fair.c static int wake_affine(struct sched_domain *sd, struct task_struct *p, · line 6120
Quoted 6 lines
kernel/sched/fair.c +1
-1schedstat_inc (p->stats.nr_wakeups_affine_attempts); -if (target == nr_cpumask_bits) +if (target != this_cpu)return prev_cpu;schedstat_inc (sd->ttwu_move_affine);This not only changes the accounting but also the placement, no?
No, it should only change the accounting. wake_affine() still returns prev_cpu if target equals to prev_cpu or nr_cpumask_bits, the same behavior as before.
Libo
Quoted 34 lines · Libo Chen
Quoted 30 lines · Peter Zijlstra
Quoted 29 lines · Libo Chen
There are scenarios where non-affine wakeups are incorrectly counted as affine wakeups by schedstats.
When
wake_affine_idle()returnsprev_cpuwhich doesn't equal tonr_cpumask_bits, it will slip through the check: target ==nr_cpumask_bitsinwake_affine()and be counted as if target ==this_cpuin schedstats.Replace target ==
nr_cpumask_bitswith target !=this_cputo make sure affine wakeups are accurately tallied.Fixes 806486c377e33 (sched/fair: Do not migrate if the prev_cpu is idle)Suggested-by Daniel Jordan <[email protected]>Signed-off-by Libo Chen <[email protected]>--- kernel/sched/fair.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)kernel/sched/fair.c +1
-1@@ -6114,7 +6114,7 @@ static int wake_affine(struct sched_domain *sd, struct task_struct *p, target =wake_affine_weight (sd, p, this_cpu, prev_cpu, sync);schedstat_inc (p->stats.nr_wakeups_affine_attempts); -if (target == nr_cpumask_bits) +if (target != this_cpu)return prev_cpu;
Quoted 5 lines
Quoted 2 lines
kernel/sched/fair.c static int wake_affine(struct sched_domain *sd, struct task_struct *p, · line 6120
Quoted 1 line
kernel/sched/fair.c +0
-0schedstat_inc (sd->ttwu_move_affine);This not only changes the accounting but also the placement, no?
No, it should only change the accounting.
wake_affine()still returnsprev_cpuif target equals toprev_cpuornr_cpumask_bits, the same behavior as before.
Looks good to me
Quoted 3 lines
Libo
Hi Peter,
A gentle ping~ Vincent has signed it off. Let me know what else I should do for this patch.
Libo
Quoted 38 lines · Libo Chen
Quoted 18 lines · Peter Zijlstra
Quoted 17 lines · Libo Chen
There are scenarios where non-affine wakeups are incorrectly counted as affine wakeups by schedstats.
When
wake_affine_idle()returnsprev_cpuwhich doesn't equal tonr_cpumask_bits, it will slip through the check: target ==nr_cpumask_bitsinwake_affine()and be counted as if target ==this_cpuin schedstats.Replace target ==
nr_cpumask_bitswith target !=this_cputo make sure affine wakeups are accurately tallied.Fixes 806486c377e33 (sched/fair: Do not migrate if the prev_cpu is idle)Suggested-by Daniel Jordan <[email protected]>Signed-off-by Libo Chen <[email protected]>---
kernel/sched/fair.c| 2 +-1 file changed, 1 insertion(+), 1 deletion(-)Quoted 13 lines
kernel/sched/fair.c static int wake_affine(struct sched_domain · line 6119
Quoted 12 lines
kernel/sched/fair.c +1
-1@@ -6114,7 +6114,7 @@ static int wake_affine(struct sched_domain *sd, struct task_struct *p, target = wake_affine_weight(sd, p, this_cpu, prev_cpu, sync); schedstat_inc(p->stats.nr_wakeups_affine_attempts); - if (target == nr_cpumask_bits) + if (target != this_cpu) return prev_cpu; schedstat_inc(sd->ttwu_move_affine);This not only changes the accounting but also the placement, no?
No, it should only change the accounting.
wake_affine()still returnsprev_cpuif target equals toprev_cpuornr_cpumask_bits, the same behavior as before.Libo
Quoted 46 lines · Libo Chen
Hi Peter,
A gentle ping~ Vincent has signed it off. Let me know what else I should do for this patch.
Libo
Quoted 35 lines · Libo Chen
Quoted 18 lines · Peter Zijlstra
Quoted 17 lines · Libo Chen
There are scenarios where non-affine wakeups are incorrectly counted as affine wakeups by schedstats.
When
wake_affine_idle()returnsprev_cpuwhich doesn't equal tonr_cpumask_bits, it will slip through the check: target ==nr_cpumask_bitsinwake_affine()and be counted as if target ==this_cpuin schedstats.Replace target ==
nr_cpumask_bitswith target !=this_cputo make sure affine wakeups are accurately tallied.Fixes 806486c377e33 (sched/fair: Do not migrate if the prev_cpu is idle)Suggested-by Daniel Jordan <[email protected]>Signed-off-by Libo Chen <[email protected]>--- kernel/sched/fair.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)Quoted 12 lines
kernel/sched/fair.c static int wake_affine(struct sched_domain *sd, struct task_struct *p, · line 6118
Quoted 11 lines
kernel/sched/fair.c +1
-1@@ -6114,7 +6114,7 @@ static int wake_affine(struct sched_domain *sd, struct task_struct *p, target =wake_affine_weight (sd, p, this_cpu, prev_cpu, sync);schedstat_inc (p->stats.nr_wakeups_affine_attempts); -if (target == nr_cpumask_bits) +if (target != this_cpu)return prev_cpu;schedstat_inc (sd->ttwu_move_affine);This not only changes the accounting but also the placement, no?
No, it should only change the accounting.
wake_affine()still returnsprev_cpuif target equals toprev_cpuornr_cpumask_bits, the same behavior as before.
Hi Peter,
A second ping in case you missed the first one, what else should I do to get this fix in?
Libo
Quoted 3 lines
Quoted 2 lines
Libo
Quoted 22 lines · Libo Chen
There are scenarios where non-affine wakeups are incorrectly counted as affine wakeups by schedstats.
When
wake_affine_idle()returnsprev_cpuwhich doesn't equal tonr_cpumask_bits, it will slip through the check: target ==nr_cpumask_bitsinwake_affine()and be counted as if target ==this_cpuin schedstats.Replace target ==
nr_cpumask_bitswith target !=this_cputo make sure affine wakeups are accurately tallied.Fixes 806486c377e33 (sched/fair: Do not migrate if the prev_cpu is idle)Suggested-by Daniel Jordan <[email protected]>Signed-off-by Libo Chen <[email protected]>--- kernel/sched/fair.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)kernel/sched/fair.c +0
-0
kernel/sched/fair.c static int wake_affine(struct sched_domain *sd, struct task_struct *p, · line 6118
Quoted 6 lines
kernel/sched/fair.c +1
-1@@ -6114,7 +6114,7 @@ static int wake_affine(struct sched_domain *sd, struct task_struct *p, target =wake_affine_weight (sd, p, this_cpu, prev_cpu, sync);schedstat_inc (p->stats.nr_wakeups_affine_attempts); -if (target == nr_cpumask_bits) +if (target != this_cpu)return prev_cpu;
This seems to be the right thing to do. However,..
if this_cpu and prev_cpu were in the same LLC and we pick prev_cpu, technically is it still not an affine wakeup?
Quoted 5 lines
kernel/sched/fair.c +0
-0schedstat_inc (sd->ttwu_move_affine);Signature · 1 line
2.31.1
--
Thanks and Regards
gautham.
Quoted 32 lines · Gautham R. Shenoy
Quoted 22 lines · Libo Chen
There are scenarios where non-affine wakeups are incorrectly counted as affine wakeups by schedstats.
When
wake_affine_idle()returnsprev_cpuwhich doesn't equal tonr_cpumask_bits, it will slip through the check: target ==nr_cpumask_bitsinwake_affine()and be counted as if target ==this_cpuin schedstats.Replace target ==
nr_cpumask_bitswith target !=this_cputo make sure affine wakeups are accurately tallied.Fixes 806486c377e33 (sched/fair: Do not migrate if the prev_cpu is idle)Suggested-by Daniel Jordan <[email protected]>Signed-off-by Libo Chen <[email protected]>--- kernel/sched/fair.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)kernel/sched/fair.c +0
-0kernel/sched/fair.c static int wake_affine(struct sched_domain *sd, struct task_struct *p, · line 6118
Quoted 6 lines
kernel/sched/fair.c +1
-1@@ -6114,7 +6114,7 @@ static int wake_affine(struct sched_domain *sd, struct task_struct *p, target =wake_affine_weight (sd, p, this_cpu, prev_cpu, sync);schedstat_inc (p->stats.nr_wakeups_affine_attempts); -if (target == nr_cpumask_bits) +if (target != this_cpu)return prev_cpu;This seems to be the right thing to do. However,..
Quoted 3 lines
if
this_cpuandprev_cpuwere in the same LLC and we pickprev_cpu, technically is it still not an affine wakeup?
I think schedstats like ttwu_move_affine/ttwu_wake_remote is defined within a sched domain, so if the waking CPU and the previous CPU are in the same MC domain, then picking the previous CPU is a remote wakeup within that MC. If the two candidate CPUs are from two different NUMA nodes, then picking the waking CPU is an affine wakeup within that NUMA domain. Correct me if I am wrong, this definition is consistent across all levels of sched domains.
But I do understand that when two candidate CPUs are within an LLC,
a) all the fast-path wakeups should be affine wakeups if your definition of an affine wakeup is a wakeup to the same LLC of the waker.
b) select_idle_sibling() may pick a CPU in that LLC other than the two candidate CPUs which makes the affine/remote stats here useless even if we are consistent with ttwu_move_affine/ttwu_wake_remote definition.
I personally think it's just too much trouble to add additional code in the kernel to, let's say, treat all wakeups within an LLC as ttwu_move_affine. It's a lot easier to do that when you process schedstats data, whether you want to treat all wakeups in LLC domains as affine wakeups or ignore ttwu_move_affine/ttwu_wake_remote stats from LLC domains.
Quoted 8 lines
Quoted 5 lines
kernel/sched/fair.c +0
-0schedstat_inc (sd->ttwu_move_affine);Signature · 1 line
2.31.1--
Thanks and Regards
gautham.
Gautham R. Shenoy
Reviewed-byHello Libo,
Quoted 1 line · Libo Chen
Sorry, looks like this message got burried under the pile.
Quoted 36 lines · Gautham R. Shenoy
Quoted 31 lines · Libo Chen
Quoted 22 lines
There are scenarios where non-affine wakeups are incorrectly counted as affine wakeups by schedstats.
When
wake_affine_idle()returnsprev_cpuwhich doesn't equal tonr_cpumask_bits, it will slip through the check: target ==nr_cpumask_bitsinwake_affine()and be counted as if target ==this_cpuin schedstats.Replace target ==
nr_cpumask_bitswith target !=this_cputo make sure affine wakeups are accurately tallied.Fixes 806486c377e33 (sched/fair: Do not migrate if the prev_cpu is idle)Suggested-by Daniel Jordan <[email protected]>Signed-off-by Libo Chen <[email protected]>--- kernel/sched/fair.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)kernel/sched/fair.c +0
-0kernel/sched/fair.c static int wake_affine(struct sched_domain *sd, struct task_struct *p, · line 6117
Quoted 5 lines
kernel/sched/fair.c +1
-1@@ -6114,7 +6114,7 @@ static int wake_affine(struct sched_domain *sd, struct task_struct *p, target =wake_affine_weight (sd, p, this_cpu, prev_cpu, sync);schedstat_inc (p->stats.nr_wakeups_affine_attempts); -if (target == nr_cpumask_bits) +if (target != this_cpu)return prev_cpu;This seems to be the right thing to do. However,..
Quoted 3 lines
if
this_cpuandprev_cpuwere in the same LLC and we pickprev_cpu, technically is it still not an affine wakeup?
Quoted 7 lines
I think schedstats like ttwu_move_affine/ttwu_wake_remote is defined within a sched domain, so if the waking CPU and the previous CPU are in the same MC domain, then picking the previous CPU is a remote wakeup
within that MC. If the two candidate CPUs are from two different NUMA nodes, then picking the waking CPU is an affine wakeup within that NUMA domain. Correct me if I am wrong, this definition is consistent across all levels of sched domains.
Yes, the definition of ttwu_wake_remote in the lowest sched-domain containing both the prev_cpu and this_cpu, is target_cpu != this_cpu. This is fairly unambiguous.
From the code, the definition of an ttwu_move_affine is to capture an _attempt_ to chose the this_cpu as the target_cpu in the lowest sched-domain containing both prev CPU and this_cpu. It is merely an attempt since the actual target_CPU is selected by SIS and could be any idle CPU in the LLC of the prev/this_cpu.
ttwu_move_affine makes sense for sched-domains higher than the LLC domain since we move the task to the LLC of this_cpu away from the LLC of the prev_cpu (This is possible on AMD processors which contains multiple LLC domains within a NUMA node). Having given it some more thought, I am not sure how to interpret this metric for the LLC domain and lower ones, since the eventual target CPU may not even be "closer" to this_cpu.
Quoted 8 lines
But I do understand that when two candidate CPUs are within an LLC,
a) all the fast-path wakeups should be affine wakeups if your definition of an affine wakeup is a wakeup to the same LLC of the waker.
b)select_idle_sibling()may pick a CPU in that LLC other than the two candidate CPUs which makes the affine/remote stats here useless even if we are consistent with ttwu_move_affine/ttwu_wake_remote definition.
Fair enough.
Quoted 6 lines
I personally think it's just too much trouble to add additional code in the kernel to, let's say, treat all wakeups within an LLC as
ttwu_move_affine. It's a lot easier to do that when you process schedstats data, whether you want to treat all wakeups in LLC domains as affine wakeups or ignore ttwu_move_affine/ttwu_wake_remote stats from LLC domains.
I agree.
I think that having your fix is the right thing, since currently the move_affine data in schedstats isn't accurate when wake_affine_idle() or wake_affine_weight() picks the prev_cpu, especially when prev_cpu and this_cpu are in sched-domains higher than the LLC. Thus today we overcount affine wakeups which is incorrect.
So,
--
Thanks and Regards
gautham.
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 39afe5d6fc59237ff7738bf3ede5a8856822d59d Gitweb: https://git.kernel.org/tip/39afe5d6fc59237ff7738bf3ede5a8856822d59d Author: Libo Chen <[email protected]> AuthorDate: Wed, 10 Aug 2022 15:33:13 -07:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Wed, 05 Apr 2023 09:58:48 +02:00
sched/fair: Fix inaccurate tally of ttwu_move_affine
There are scenarios where non-affine wakeups are incorrectly counted as affine wakeups by schedstats.
When wake_affine_idle() returns prev_cpu which doesn't equal to nr_cpumask_bits, it will slip through the check: target == nr_cpumask_bits in wake_affine() and be counted as if target == this_cpu in schedstats.
Replace target == nr_cpumask_bits with target != this_cpu to make sure affine wakeups are accurately tallied.
--- kernel/sched/fair.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/sched/fair.c +1 -1
@@ -6582,7 +6582,7 @@ static int wake_affine(struct sched_domain *sd, struct task_struct *p, target =wake_affine_weight (sd, p, this_cpu, prev_cpu, sync);schedstat_inc (p->stats.nr_wakeups_affine_attempts); -if (target == nr_cpumask_bits) +if (target != this_cpu)return prev_cpu;schedstat_inc (sd->ttwu_move_affine);