Skip to contentSkip to conversation

Reduce SIS scanning

PATCH Applied Mel Gorman 3 participants 6 messages Aug 4, 2021 – Aug 5, 2021 Scheduler

lore
Share & tools
lore

Mel Gorman

author
#0 lore raw

cover Reduce SIS scanning

This is the first two patches from "Modify and/or delete SIS_PROP" and focues on improving the hit rate for recent_used_cpu and avoids rescanning the target CPU when it's known to be busy. These are relatively simply patches in comparison to the full series which could be a regression magnet and that series had at least one mistake in it.

In general, the usual suspects showed mostly small gains with a few exceptions. Usual suspects were NAS, hackbench, schbench (Facebook latency-sensitive workload), perf pipe, kernel building, git test suite, dbench and redis.

From here, the next obvious candidates are either improving has_idle_cores or continuing to try remove/improve SIS_PROP.

 kernel/sched/fair.c | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)
Signature · 1 line
2.31.1
author
#1 lore raw

PATCH 1/2 sched/fair: Use prev instead of new target as recent_used_cpu

After select_idle_sibling, p->recent_used_cpu is set to the new target. However on the next wakeup, prev will be the same as recent_used_cpu unless the load balancer has moved the task since the last wakeup. It still works, but is less efficient than it could be. This patch preserves recent_used_cpu for longer.

The impact on SIS efficiency is tiny so the SIS statistic patches were used to track the hit rate for using recent_used_cpu. With perf bench pipe on a 2-socket Cascadelake machine, the hit rate went from 57.14% to 85.32%. For more intensive wakeup loads like hackbench, the hit rate is almost negligible but rose from 0.21% to 6.64%. For scaling loads like tbench, the hit rate goes from almost 0% to 25.42% overall. Broadly speaking, on tbench, the success rate is much higher for lower thread counts and drops to almost 0 as the workload scales to towards saturation.

---
 kernel/sched/fair.c | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)
kernel/sched/fair.c +1 -3
index 44c452072a1b..8ad7666f387c 100644
@@ -6376,6 +6376,7 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
 
 	/* Check a recently used CPU as a potential idle candidate: */
 	recent_used_cpu = p->recent_used_cpu;
+	p->recent_used_cpu = prev;
 	if (recent_used_cpu != prev &&
 	    recent_used_cpu != target &&
 	    cpus_share_cache(recent_used_cpu, target) &&
@@ -6902,9 +6903,6 @@ select_task_rq_fair(struct task_struct *p, int prev_cpu, int wake_flags)
 	} else if (wake_flags & WF_TTWU) { /* XXX always ? */
 		/* Fast path */
 		new_cpu = select_idle_sibling(p, prev_cpu, new_cpu);
-
-		if (want_affine)
-			current->recent_used_cpu = cpu;
 	}
 	rcu_read_unlock();
 
Signature · 1 line
2.31.1
merge bot Applied
#2 lore raw

[tip: sched/core] sched/fair: Use prev instead of new target as recent_used_cpu

The following commit has been merged into the sched/core branch of tip:

The following commit has been merged into the sched/core branch of tip:

Commit-ID:     89aafd67f28c9e3b725aa30b44b7f61ad3e348ce
Gitweb:        https://git.kernel.org/tip/89aafd67f28c9e3b725aa30b44b7f61ad3e348ce
Author:        Mel Gorman <[email protected]>
AuthorDate:    Wed, 04 Aug 2021 12:58:56 +01:00
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Wed, 04 Aug 2021 15:16:44 +02:00

sched/fair: Use prev instead of new target as recent_used_cpu

After select_idle_sibling, p->recent_used_cpu is set to the new target. However on the next wakeup, prev will be the same as recent_used_cpu unless the load balancer has moved the task since the last wakeup. It still works, but is less efficient than it could be. This patch preserves recent_used_cpu for longer.

The impact on SIS efficiency is tiny so the SIS statistic patches were used to track the hit rate for using recent_used_cpu. With perf bench pipe on a 2-socket Cascadelake machine, the hit rate went from 57.14% to 85.32%. For more intensive wakeup loads like hackbench, the hit rate is almost negligible but rose from 0.21% to 6.64%. For scaling loads like tbench, the hit rate goes from almost 0% to 25.42% overall. Broadly speaking, on tbench, the success rate is much higher for lower thread counts and drops to almost 0 as the workload scales to towards saturation.

---
 kernel/sched/fair.c | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)
kernel/sched/fair.c +1 -3
index 13c3fd4..7ee394b 100644
@@ -6347,6 +6347,7 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
 
 	/* Check a recently used CPU as a potential idle candidate: */
 	recent_used_cpu = p->recent_used_cpu;
+	p->recent_used_cpu = prev;
 	if (recent_used_cpu != prev &&
 	    recent_used_cpu != target &&
 	    cpus_share_cache(recent_used_cpu, target) &&
@@ -6873,9 +6874,6 @@ select_task_rq_fair(struct task_struct *p, int prev_cpu, int wake_flags)
 	} else if (wake_flags & WF_TTWU) { /* XXX always ? */
 		/* Fast path */
 		new_cpu = select_idle_sibling(p, prev_cpu, new_cpu);
-
-		if (want_affine)
-			current->recent_used_cpu = cpu;
 	}
 	rcu_read_unlock();
 
author
#3 lore raw

PATCH 2/2 sched/fair: Avoid a second scan of target in select_idle_cpu

In reply to Mel Gorman

When select_idle_cpu starts scanning for an idle CPU, it starts with a target CPU that has already been checked by select_idle_sibling. This patch starts with the next CPU instead.

---
 kernel/sched/fair.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/sched/fair.c +1 -1
index 8ad7666f387c..f412b4504378 100644
@@ -6249,7 +6249,7 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, bool
 		time = cpu_clock(this);
 	}
 
-	for_each_cpu_wrap(cpu, cpus, target) {
+	for_each_cpu_wrap(cpu, cpus, target + 1) {
 		if (has_idle_core) {
 			i = select_idle_core(p, cpu, cpus, &idle_cpu);
 			if ((unsigned int)i < nr_cpumask_bits)
Signature · 1 line
2.31.1
merge bot Applied
#4 lore raw

[tip: sched/core] sched/fair: Avoid a second scan of target in select_idle_cpu

The following commit has been merged into the sched/core branch of tip:

The following commit has been merged into the sched/core branch of tip:

Commit-ID:     56498cfb045d7147cdcba33795d19429afcd1d00
Gitweb:        https://git.kernel.org/tip/56498cfb045d7147cdcba33795d19429afcd1d00
Author:        Mel Gorman <[email protected]>
AuthorDate:    Wed, 04 Aug 2021 12:58:57 +01:00
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Wed, 04 Aug 2021 15:16:44 +02:00

sched/fair: Avoid a second scan of target in select_idle_cpu

When select_idle_cpu starts scanning for an idle CPU, it starts with a target CPU that has already been checked by select_idle_sibling. This patch starts with the next CPU instead.

---
 kernel/sched/fair.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/sched/fair.c +1 -1
index 7ee394b..47a0fbf 100644
@@ -6220,7 +6220,7 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, bool 
 		time = cpu_clock(this);
 	}
 
-	for_each_cpu_wrap(cpu, cpus, target) {
+	for_each_cpu_wrap(cpu, cpus, target + 1) {
 		if (has_idle_core) {
 			i = select_idle_core(p, cpu, cpus, &idle_cpu);
 			if ((unsigned int)i < nr_cpumask_bits)
maintainer
#5 lore raw
In reply to Mel Gorman
Quoted 6 lines · Mel Gorman

This is the first two patches from "Modify and/or delete SIS_PROP" and focues on improving the hit rate for recent_used_cpu and avoids rescanning the target CPU when it's known to be busy. These are relatively simply patches in comparison to the full series which could be a regression magnet and that series had at least one mistake in it.

Quoted 7 lines

In general, the usual suspects showed mostly small gains with a few exceptions. Usual suspects were NAS, hackbench, schbench (Facebook latency-sensitive workload), perf pipe, kernel building, git test suite, dbench and redis.

From here, the next obvious candidates are either improving has_idle_cores or continuing to try remove/improve SIS_PROP.

Thanks!