Skip to contentSkip to conversation

sched/fair: Fix inaccurate tally of ttwu_move_affine

PATCH Applied Libo Chen 5 participants 10 messages Aug 10, 2022 – Apr 6, 2023 Scheduler

lore
Share & tools
lore

Libo Chen

author
#0 lore raw

PATCH 1/1 sched/fair: Fix inaccurate tally of ttwu_move_affine

There are scenarios where non-affine wakeups are incorrectly counted as affine wakeups by schedstats.

When wake_affine_idle() returns prev_cpu which doesn't equal to nr_cpumask_bits, it will slip through the check: target == nr_cpumask_bits in wake_affine() and be counted as if target == this_cpu in schedstats.

Replace target == nr_cpumask_bits with target != this_cpu to make sure affine wakeups are accurately tallied.

Fixes 806486c377e33 (sched/fair: Do not migrate if the prev_cpu is idle)
---
 kernel/sched/fair.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/sched/fair.c +1 -1
index da388657d5ac..b179da4f8105 100644
@@ -6114,7 +6114,7 @@ static int wake_affine(struct sched_domain *sd, struct task_struct *p,
 		target = wake_affine_weight(sd, p, this_cpu, prev_cpu, sync);
 
 	schedstat_inc(p->stats.nr_wakeups_affine_attempts);
-	if (target == nr_cpumask_bits)
+	if (target != this_cpu)
 		return prev_cpu;
 
 	schedstat_inc(sd->ttwu_move_affine);
Signature · 1 line
2.31.1
maintainer
#1 lore raw
Quoted 22 lines · Libo Chen

There are scenarios where non-affine wakeups are incorrectly counted as affine wakeups by schedstats.

When wake_affine_idle() returns prev_cpu which doesn't equal to nr_cpumask_bits, it will slip through the check: target == nr_cpumask_bits in wake_affine() and be counted as if target == this_cpu in schedstats.

Replace target == nr_cpumask_bits with target != this_cpu to make sure affine wakeups are accurately tallied.

Fixes 806486c377e33 (sched/fair: Do not migrate if the prev_cpu is idle)
---
 kernel/sched/fair.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/sched/fair.c +0 -0
index da388657d5ac..b179da4f8105 100644

kernel/sched/fair.c static int wake_affine(struct sched_domain *sd, struct task_struct *p, · line 6120

Quoted 8 lines
kernel/sched/fair.c +1 -1
@@ -6114,7 +6114,7 @@ static int wake_affine(struct sched_domain *sd, struct task_struct *p,
 		target = wake_affine_weight(sd, p, this_cpu, prev_cpu, sync);
 
 	schedstat_inc(p->stats.nr_wakeups_affine_attempts);
-	if (target == nr_cpumask_bits)
+	if (target != this_cpu)
 		return prev_cpu;
 
 	schedstat_inc(sd->ttwu_move_affine);

This not only changes the accounting but also the placement, no?

author
#2 lore raw
Quoted 25 lines · Peter Zijlstra
Quoted 24 lines · Libo Chen

There are scenarios where non-affine wakeups are incorrectly counted as affine wakeups by schedstats.

When wake_affine_idle() returns prev_cpu which doesn't equal to nr_cpumask_bits, it will slip through the check: target == nr_cpumask_bits in wake_affine() and be counted as if target == this_cpu in schedstats.

Replace target == nr_cpumask_bits with target != this_cpu to make sure affine wakeups are accurately tallied.

Fixes 806486c377e33 (sched/fair: Do not migrate if the prev_cpu is idle)
---
  kernel/sched/fair.c | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
kernel/sched/fair.c +0 -0
index da388657d5ac..b179da4f8105 100644
@@ -6114,7 +6114,7 @@ static int wake_affine(struct sched_domain *sd, struct task_struct *p,
  		target = wake_affine_weight(sd, p, this_cpu, prev_cpu, sync);
 
Quoted 7 lines

kernel/sched/fair.c static int wake_affine(struct sched_domain *sd, struct task_struct *p, · line 6120

Quoted 6 lines
kernel/sched/fair.c +1 -1
  	schedstat_inc(p->stats.nr_wakeups_affine_attempts);
-	if (target == nr_cpumask_bits)
+	if (target != this_cpu)
  		return prev_cpu;
 
  	schedstat_inc(sd->ttwu_move_affine);

This not only changes the accounting but also the placement, no?

No, it should only change the accounting. wake_affine() still returns prev_cpu if target equals to prev_cpu or nr_cpumask_bits, the same behavior as before.

Libo

maintainer Reviewed-by
#3 lore raw
Quoted 34 lines · Libo Chen
Quoted 30 lines · Peter Zijlstra
Quoted 29 lines · Libo Chen

There are scenarios where non-affine wakeups are incorrectly counted as affine wakeups by schedstats.

When wake_affine_idle() returns prev_cpu which doesn't equal to nr_cpumask_bits, it will slip through the check: target == nr_cpumask_bits in wake_affine() and be counted as if target == this_cpu in schedstats.

Replace target == nr_cpumask_bits with target != this_cpu to make sure affine wakeups are accurately tallied.

Fixes 806486c377e33 (sched/fair: Do not migrate if the prev_cpu is idle)
---
  kernel/sched/fair.c | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
kernel/sched/fair.c +1 -1
index da388657d5ac..b179da4f8105 100644
@@ -6114,7 +6114,7 @@ static int wake_affine(struct sched_domain *sd, struct task_struct *p,
             target = wake_affine_weight(sd, p, this_cpu, prev_cpu, sync);
 
     schedstat_inc(p->stats.nr_wakeups_affine_attempts);
-    if (target == nr_cpumask_bits)
+    if (target != this_cpu)
             return prev_cpu;
 
Quoted 5 lines
Quoted 2 lines

kernel/sched/fair.c static int wake_affine(struct sched_domain *sd, struct task_struct *p, · line 6120

Quoted 1 line
kernel/sched/fair.c +0 -0
     schedstat_inc(sd->ttwu_move_affine);

This not only changes the accounting but also the placement, no?

No, it should only change the accounting. wake_affine() still returns prev_cpu if target equals to prev_cpu or nr_cpumask_bits, the same behavior as before.

Looks good to me

Quoted 3 lines

Libo

author
#4 lore raw
In reply to Libo Chen

Hi Peter,

A gentle ping~ Vincent has signed it off. Let me know what else I should do for this patch.

Libo

Quoted 38 lines · Libo Chen
Quoted 18 lines · Peter Zijlstra
Quoted 17 lines · Libo Chen

There are scenarios where non-affine wakeups are incorrectly counted as affine wakeups by schedstats.

When wake_affine_idle() returns prev_cpu which doesn't equal to nr_cpumask_bits, it will slip through the check: target == nr_cpumask_bits in wake_affine() and be counted as if target == this_cpu in schedstats.

Replace target == nr_cpumask_bits with target != this_cpu to make sure affine wakeups are accurately tallied.

Fixes 806486c377e33 (sched/fair: Do not migrate if the prev_cpu is idle)

---
  kernel/sched/fair.c | 2 +-

  1 file changed, 1 insertion(+), 1 deletion(-)
Quoted 13 lines

kernel/sched/fair.c static int wake_affine(struct sched_domain · line 6119

Quoted 12 lines
kernel/sched/fair.c +1 -1
index da388657d5ac..b179da4f8105 100644
@@ -6114,7 +6114,7 @@ static int wake_affine(struct sched_domain 
 *sd, struct task_struct *p,
           target = wake_affine_weight(sd, p, this_cpu, prev_cpu, sync);
         schedstat_inc(p->stats.nr_wakeups_affine_attempts);
-    if (target == nr_cpumask_bits)
+    if (target != this_cpu)
           return prev_cpu;
         schedstat_inc(sd->ttwu_move_affine);

This not only changes the accounting but also the placement, no?

No, it should only change the accounting. wake_affine() still returns prev_cpu if target equals to prev_cpu or nr_cpumask_bits, the same behavior as before.

Libo

author
#5 lore raw
Quoted 46 lines · Libo Chen

Hi Peter,

A gentle ping~ Vincent has signed it off. Let me know what else I should do for this patch.

Libo

Quoted 35 lines · Libo Chen
Quoted 18 lines · Peter Zijlstra
Quoted 17 lines · Libo Chen

There are scenarios where non-affine wakeups are incorrectly counted as affine wakeups by schedstats.

When wake_affine_idle() returns prev_cpu which doesn't equal to nr_cpumask_bits, it will slip through the check: target == nr_cpumask_bits in wake_affine() and be counted as if target == this_cpu in schedstats.

Replace target == nr_cpumask_bits with target != this_cpu to make sure affine wakeups are accurately tallied.

Fixes 806486c377e33 (sched/fair: Do not migrate if the prev_cpu is idle)
---
  kernel/sched/fair.c | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
Quoted 12 lines

kernel/sched/fair.c static int wake_affine(struct sched_domain *sd, struct task_struct *p, · line 6118

Quoted 11 lines
kernel/sched/fair.c +1 -1
index da388657d5ac..b179da4f8105 100644
@@ -6114,7 +6114,7 @@ static int wake_affine(struct sched_domain *sd, struct task_struct *p,
          target = wake_affine_weight(sd, p, this_cpu, prev_cpu, sync);
        schedstat_inc(p->stats.nr_wakeups_affine_attempts);
-    if (target == nr_cpumask_bits)
+    if (target != this_cpu)
          return prev_cpu;
        schedstat_inc(sd->ttwu_move_affine);

This not only changes the accounting but also the placement, no?

No, it should only change the accounting. wake_affine() still returns prev_cpu if target equals to prev_cpu or nr_cpumask_bits, the same behavior as before.

Hi Peter,

A second ping in case you missed the first one, what else should I do to get this fix in?

Libo

Quoted 3 lines
Quoted 2 lines

Libo

#6 lore raw
In reply to Libo Chen
Quoted 22 lines · Libo Chen

There are scenarios where non-affine wakeups are incorrectly counted as affine wakeups by schedstats.

When wake_affine_idle() returns prev_cpu which doesn't equal to nr_cpumask_bits, it will slip through the check: target == nr_cpumask_bits in wake_affine() and be counted as if target == this_cpu in schedstats.

Replace target == nr_cpumask_bits with target != this_cpu to make sure affine wakeups are accurately tallied.

Fixes 806486c377e33 (sched/fair: Do not migrate if the prev_cpu is idle)
---
 kernel/sched/fair.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/sched/fair.c +0 -0
index da388657d5ac..b179da4f8105 100644

kernel/sched/fair.c static int wake_affine(struct sched_domain *sd, struct task_struct *p, · line 6118

Quoted 6 lines
kernel/sched/fair.c +1 -1
@@ -6114,7 +6114,7 @@ static int wake_affine(struct sched_domain *sd, struct task_struct *p,
 		target = wake_affine_weight(sd, p, this_cpu, prev_cpu, sync);
 
 	schedstat_inc(p->stats.nr_wakeups_affine_attempts);
-	if (target == nr_cpumask_bits)
+	if (target != this_cpu)
 		return prev_cpu;

This seems to be the right thing to do. However,..

if this_cpu and prev_cpu were in the same LLC and we pick prev_cpu, technically is it still not an affine wakeup?

Quoted 5 lines
kernel/sched/fair.c +0 -0
 	schedstat_inc(sd->ttwu_move_affine);
Signature · 1 line
2.31.1

--
Thanks and Regards
gautham.

author
#7 lore raw
Quoted 32 lines · Gautham R. Shenoy
Quoted 22 lines · Libo Chen

There are scenarios where non-affine wakeups are incorrectly counted as affine wakeups by schedstats.

When wake_affine_idle() returns prev_cpu which doesn't equal to nr_cpumask_bits, it will slip through the check: target == nr_cpumask_bits in wake_affine() and be counted as if target == this_cpu in schedstats.

Replace target == nr_cpumask_bits with target != this_cpu to make sure affine wakeups are accurately tallied.

Fixes 806486c377e33 (sched/fair: Do not migrate if the prev_cpu is idle)
---
  kernel/sched/fair.c | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
kernel/sched/fair.c +0 -0
index da388657d5ac..b179da4f8105 100644

kernel/sched/fair.c static int wake_affine(struct sched_domain *sd, struct task_struct *p, · line 6118

Quoted 6 lines
kernel/sched/fair.c +1 -1
@@ -6114,7 +6114,7 @@ static int wake_affine(struct sched_domain *sd, struct task_struct *p,
  		target = wake_affine_weight(sd, p, this_cpu, prev_cpu, sync);
 
  	schedstat_inc(p->stats.nr_wakeups_affine_attempts);
-	if (target == nr_cpumask_bits)
+	if (target != this_cpu)
  		return prev_cpu;

This seems to be the right thing to do. However,..

Quoted 3 lines

if this_cpu and prev_cpu were in the same LLC and we pick prev_cpu, technically is it still not an affine wakeup?

I think schedstats like ttwu_move_affine/ttwu_wake_remote is defined within a sched domain, so if the waking CPU and the previous CPU are in the same MC domain, then picking the previous CPU is a remote wakeup within that MC. If the two candidate CPUs are from two different NUMA nodes, then picking the waking CPU is an affine wakeup within that NUMA domain. Correct me if I am wrong, this definition is consistent across all levels of sched domains.

But I do understand that when two candidate CPUs are within an LLC,
    a) all the fast-path wakeups should be affine wakeups if your definition of an affine wakeup is a wakeup to the same LLC of the waker.
    b) select_idle_sibling() may pick a CPU in that LLC other than the two candidate CPUs which makes the affine/remote stats here useless even if we are consistent with ttwu_move_affine/ttwu_wake_remote definition.

I personally think it's just too much trouble to add additional code in the kernel to, let's say, treat all wakeups within an LLC as ttwu_move_affine. It's a lot easier to do that when you process schedstats data, whether you want to treat all wakeups in LLC domains as affine wakeups or ignore ttwu_move_affine/ttwu_wake_remote stats from LLC domains.

Quoted 8 lines
Quoted 5 lines
kernel/sched/fair.c +0 -0
  	schedstat_inc(sd->ttwu_move_affine);
Signature · 1 line
2.31.1

--
Thanks and Regards
gautham.

Reviewed-by
#8 lore raw

Hello Libo,

Quoted 1 line · Libo Chen

Sorry, looks like this message got burried under the pile.

Quoted 36 lines · Gautham R. Shenoy
Quoted 31 lines · Libo Chen
Quoted 22 lines

There are scenarios where non-affine wakeups are incorrectly counted as affine wakeups by schedstats.

When wake_affine_idle() returns prev_cpu which doesn't equal to nr_cpumask_bits, it will slip through the check: target == nr_cpumask_bits in wake_affine() and be counted as if target == this_cpu in schedstats.

Replace target == nr_cpumask_bits with target != this_cpu to make sure affine wakeups are accurately tallied.

Fixes 806486c377e33 (sched/fair: Do not migrate if the prev_cpu is idle)
---
  kernel/sched/fair.c | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
kernel/sched/fair.c +0 -0
index da388657d5ac..b179da4f8105 100644

kernel/sched/fair.c static int wake_affine(struct sched_domain *sd, struct task_struct *p, · line 6117

Quoted 5 lines
kernel/sched/fair.c +1 -1
@@ -6114,7 +6114,7 @@ static int wake_affine(struct sched_domain *sd, struct task_struct *p,
  		target = wake_affine_weight(sd, p, this_cpu, prev_cpu, sync);
  	schedstat_inc(p->stats.nr_wakeups_affine_attempts);
-	if (target == nr_cpumask_bits)
+	if (target != this_cpu)
  		return prev_cpu;

This seems to be the right thing to do. However,..

Quoted 3 lines

if this_cpu and prev_cpu were in the same LLC and we pick prev_cpu, technically is it still not an affine wakeup?

Quoted 7 lines

I think schedstats like ttwu_move_affine/ttwu_wake_remote is defined within a sched domain, so if the waking CPU and the previous CPU are in the same MC domain, then picking the previous CPU is a remote wakeup
within that MC. If the two candidate CPUs are from two different NUMA nodes, then picking the waking CPU is an affine wakeup within that NUMA domain. Correct me if I am wrong, this definition is consistent across all levels of sched domains.

Yes, the definition of ttwu_wake_remote in the lowest sched-domain containing both the prev_cpu and this_cpu, is target_cpu != this_cpu. This is fairly unambiguous.

From the code, the definition of an ttwu_move_affine is to capture an _attempt_ to chose the this_cpu as the target_cpu in the lowest sched-domain containing both prev CPU and this_cpu. It is merely an attempt since the actual target_CPU is selected by SIS and could be any idle CPU in the LLC of the prev/this_cpu.

ttwu_move_affine makes sense for sched-domains higher than the LLC domain since we move the task to the LLC of this_cpu away from the LLC of the prev_cpu (This is possible on AMD processors which contains multiple LLC domains within a NUMA node). Having given it some more thought, I am not sure how to interpret this metric for the LLC domain and lower ones, since the eventual target CPU may not even be "closer" to this_cpu.

Quoted 8 lines

But I do understand that when two candidate CPUs are within an LLC,
    a) all the fast-path wakeups should be affine wakeups if your definition of an affine wakeup is a wakeup to the same LLC of the waker.
    b) select_idle_sibling() may pick a CPU in that LLC other than the two candidate CPUs which makes the affine/remote stats here useless even if we are consistent with ttwu_move_affine/ttwu_wake_remote definition.

Fair enough.

Quoted 6 lines

I personally think it's just too much trouble to add additional code in the kernel to, let's say, treat all wakeups within an LLC as ttwu_move_affine. It's a lot easier to do that when you process schedstats data, whether you want to treat all wakeups in LLC domains as affine wakeups or ignore ttwu_move_affine/ttwu_wake_remote stats from LLC domains.

I agree.

I think that having your fix is the right thing, since currently the move_affine data in schedstats isn't accurate when wake_affine_idle() or wake_affine_weight() picks the prev_cpu, especially when prev_cpu and this_cpu are in sched-domains higher than the LLC. Thus today we overcount affine wakeups which is incorrect.

So,

--
Thanks and Regards
gautham.

merge bot Applied
#9 lore raw
In reply to Libo Chen
The following commit has been merged into the sched/core branch of tip:

The following commit has been merged into the sched/core branch of tip:

Commit-ID:     39afe5d6fc59237ff7738bf3ede5a8856822d59d
Gitweb:        https://git.kernel.org/tip/39afe5d6fc59237ff7738bf3ede5a8856822d59d
Author:        Libo Chen <[email protected]>
AuthorDate:    Wed, 10 Aug 2022 15:33:13 -07:00
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Wed, 05 Apr 2023 09:58:48 +02:00

sched/fair: Fix inaccurate tally of ttwu_move_affine

There are scenarios where non-affine wakeups are incorrectly counted as affine wakeups by schedstats.

When wake_affine_idle() returns prev_cpu which doesn't equal to nr_cpumask_bits, it will slip through the check: target == nr_cpumask_bits in wake_affine() and be counted as if target == this_cpu in schedstats.

Replace target == nr_cpumask_bits with target != this_cpu to make sure affine wakeups are accurately tallied.

---
 kernel/sched/fair.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/sched/fair.c +1 -1
index bc358dc..f5da01a 100644
@@ -6582,7 +6582,7 @@ static int wake_affine(struct sched_domain *sd, struct task_struct *p,
 		target = wake_affine_weight(sd, p, this_cpu, prev_cpu, sync);
 
 	schedstat_inc(p->stats.nr_wakeups_affine_attempts);
-	if (target == nr_cpumask_bits)
+	if (target != this_cpu)
 		return prev_cpu;
 
 	schedstat_inc(sd->ttwu_move_affine);