sched: Pass affine target cpu into wake_affine

commit a03ecf08d7bbdd979d81163ea13d194fe21ad339

Author: Lin Ming <[email protected]>
Date:   Mon Jan 4 14:14:50 2010 +0800

sched: Pass affine target cpu into wake_affine

Since commit a1f84a3(sched: Check for an idle shared cache in select_task_rq_fair()), the affine target maybe adjusted to any idle cpu in cache sharing domains instead of current cpu.
But wake_affine still use current cpu to calculate load which is wrong.

This patch passes affine cpu into wake_affine.

Quoted 11 lines · Lin Ming

commit a03ecf08d7bbdd979d81163ea13d194fe21ad339

Author: Lin Ming <[email protected]>
Date:   Mon Jan 4 14:14:50 2010 +0800

sched: Pass affine target cpu into wake_affine

Since commit a1f84a3(sched: Check for an idle shared cache in select_task_rq_fair()), the affine target maybe adjusted to any idle cpu in cache sharing domains instead of current cpu.
But wake_affine still use current cpu to calculate load which is wrong.

Quoted 2 lines

This patch passes affine cpu into wake_affine.

Does this at all help with that regression?

Quoted 12 lines · Peter Zijlstra

Quoted 11 lines · Lin Ming

commit a03ecf08d7bbdd979d81163ea13d194fe21ad339

Author: Lin Ming <[email protected]>
Date:   Mon Jan 4 14:14:50 2010 +0800

sched: Pass affine target cpu into wake_affine

Since commit a1f84a3(sched: Check for an idle shared cache in select_task_rq_fair()), the affine target maybe adjusted to any idle cpu in cache sharing domains instead of current cpu.
But wake_affine still use current cpu to calculate load which is wrong.

Quoted 4 lines

Quoted 2 lines

This patch passes affine cpu into wake_affine.

Does this at all help with that regression?

No.

Quoted 16 lines · Lin Ming

Quoted 15 lines · Peter Zijlstra

Quoted 13 lines · Lin Ming

commit a03ecf08d7bbdd979d81163ea13d194fe21ad339

Author: Lin Ming <[email protected]>
Date:   Mon Jan 4 14:14:50 2010 +0800

sched: Pass affine target cpu into wake_affine

Since commit a1f84a3(sched: Check for an idle shared cache in select_task_rq_fair()), the affine target maybe adjusted to any idle cpu in cache sharing domains instead of current cpu.
But wake_affine still use current cpu to calculate load which is wrong.

This patch passes affine cpu into wake_affine.

Quoted 3 lines

Quoted 1 line

Does this at all help with that regression?

No.

crap :/

The change does look sensible though.

Quoted 17 lines · Peter Zijlstra

Quoted 16 lines · Lin Ming

Quoted 15 lines · Peter Zijlstra

Quoted 13 lines · Lin Ming

commit a03ecf08d7bbdd979d81163ea13d194fe21ad339

Author: Lin Ming <[email protected]>
Date:   Mon Jan 4 14:14:50 2010 +0800

sched: Pass affine target cpu into wake_affine

Since commit a1f84a3(sched: Check for an idle shared cache in select_task_rq_fair()), the affine target maybe adjusted to any idle cpu in cache sharing domains instead of current cpu.
But wake_affine still use current cpu to calculate load which is wrong.

This patch passes affine cpu into wake_affine.

Quoted 7 lines

Quoted 3 lines

Quoted 1 line

Does this at all help with that regression?

No.

crap :/

The change does look sensible though.

I piddled with all kinds of ways to get around calling wake_affine() entirely, and/or calling it with the affine candidate to no avail. Best result was always to do the silly looking thing, namely test the current cpu for wake affine decision, but slip in the shared cache cpu.

I bet the below helps, though there will still be cache misses, so there will still be pain for extreme switchers. Question is whether the ramp-up gain is worth it. I think yes, since it's up to 100%. Would be most excellent to find a way to know in advance when the cost will be too high, and then not go there. Same applies for doing the affinity decision every time for extreme switchers. It's expensive for those, especially so when they're pinned, but pays in the general case.

Anyway...

PREFER_SIBLING is set at the CPU domain level if you don't have power saving set, so you get to eat cache misses for each cpu, whether it's sharing a cache or not as you traverse. Lots of CPUs, LOTS of pain.

not-signed-off

include/linux/topology.h +1 -1

index 57e6357..5b81156 100644
@@ -99,7 +99,7 @@ int arch_update_cpu_topology(void);
 				| 1*SD_WAKE_AFFINE			\
 				| 1*SD_SHARE_CPUPOWER			\
 				| 0*SD_POWERSAVINGS_BALANCE		\
-				| 0*SD_SHARE_PKG_RESOURCES		\
+				| 1*SD_SHARE_PKG_RESOURCES		\
 				| 0*SD_SERIALIZE			\
 				| 0*SD_PREFER_SIBLING			\
 				,					\

kernel/sched_fair.c +1 -1

index 42ac3c9..8fe7ee8 100644
@@ -1508,7 +1508,7 @@ static int select_task_rq_fair(struct task_struct *p, int sd_flag, int wake_flag
 			 * If there's an idle sibling in this domain, make that
 			 * the wake_affine target instead of the current cpu.
 			 */
-			if (tmp->flags & SD_PREFER_SIBLING)
+			if (tmp->flags & SD_SHARE_PKG_RESOURCES)
 				target = select_idle_sibling(p, tmp, target);
 
 			if (target >= 0) {

commit a03ecf08d7bbdd979d81163ea13d194fe21ad339

Author: Lin Ming <[email protected]>
Date:   Mon Jan 4 14:14:50 2010 +0800

sched: Pass affine target cpu into wake_affine

Since commit a1f84a3(sched: Check for an idle shared cache in select_task_rq_fair()), the affine target maybe adjusted to any idle cpu in cache sharing domains instead of current cpu.
But wake_affine still use current cpu to calculate load which is wrong.

This patch passes affine cpu into wake_affine.

Signed-off-by Lin Ming <[email protected]>

---
 kernel/sched_fair.c |   29 ++++++++++++++---------------
 1 files changed, 14 insertions(+), 15 deletions(-)

kernel/sched_fair.c +14 -15

index 42ac3c9..0a6fa39 100644
@@ -1237,11 +1237,11 @@ static inline unsigned long effective_load(struct task_group *tg, int cpu,
 
 #endif
 
-static int wake_affine(struct sched_domain *sd, struct task_struct *p, int sync)
+static int wake_affine(struct sched_domain *sd, struct task_struct *p, int affine_cpu, int sync)
 {
 	struct task_struct *curr = current;
-	unsigned long this_load, load;
-	int idx, this_cpu, prev_cpu;
+	unsigned long affine_load, load;
+	int idx, prev_cpu;
 	unsigned long tl_per_task;
 	unsigned int imbalance;
 	struct task_group *tg;

@@ -1249,10 +1249,9 @@ static int wake_affine(struct sched_domain *sd, struct task_struct *p, int sync)
 	int balanced;
 
 	idx	  = sd->wake_idx;
-	this_cpu  = smp_processor_id();
 	prev_cpu  = task_cpu(p);
 	load	  = source_load(prev_cpu, idx);
-	this_load = target_load(this_cpu, idx);
+	affine_load = target_load(affine_cpu, idx);
 
 	if (sync) {
 	       if (sched_feat(SYNC_LESS) &&

@@ -1275,7 +1274,7 @@ static int wake_affine(struct sched_domain *sd, struct task_struct *p, int sync)
 		tg = task_group(current);
 		weight = current->se.load.weight;
 
-		this_load += effective_load(tg, this_cpu, -weight, -weight);
+		affine_load += effective_load(tg, affine_cpu, -weight, -weight);
 		load += effective_load(tg, prev_cpu, 0, -weight);
 	}
 

@@ -1285,16 +1284,16 @@ static int wake_affine(struct sched_domain *sd, struct task_struct *p, int sync)
 	imbalance = 100 + (sd->imbalance_pct - 100) / 2;
 
 	/*
-	 * In low-load situations, where prev_cpu is idle and this_cpu is idle
-	 * due to the sync cause above having dropped this_load to 0, we'll
+	 * In low-load situations, where prev_cpu is idle and affine_cpu is idle
+	 * due to the sync cause above having dropped affine_load to 0, we'll
 	 * always have an imbalance, but there's really nothing you can do
 	 * about that, so that's good too.
 	 *
 	 * Otherwise check if either cpus are near enough in load to allow this
-	 * task to be woken on this_cpu.
+	 * task to be woken on affine_cpu.
 	 */
-	balanced = !this_load ||
-		100*(this_load + effective_load(tg, this_cpu, weight, weight)) <=
+	balanced = !affine_load ||
+		100*(affine_load + effective_load(tg, affine_cpu, weight, weight)) <=
 		imbalance*(load + effective_load(tg, prev_cpu, 0, weight));
 
 	/*

@@ -1306,11 +1305,11 @@ static int wake_affine(struct sched_domain *sd, struct task_struct *p, int sync)
 		return 1;
 
 	schedstat_inc(p, se.nr_wakeups_affine_attempts);
-	tl_per_task = cpu_avg_load_per_task(this_cpu);
+	tl_per_task = cpu_avg_load_per_task(affine_cpu);
 
 	if (balanced ||
-	    (this_load <= load &&
-	     this_load + target_load(prev_cpu, idx) <= tl_per_task)) {
+	    (affine_load <= load &&
+	     affine_load + target_load(prev_cpu, idx) <= tl_per_task)) {
 		/*
 		 * This domain has SD_WAKE_AFFINE and
 		 * p is cache cold in this domain, and

@@ -1544,7 +1543,7 @@ static int select_task_rq_fair(struct task_struct *p, int sd_flag, int wake_flag
 			update_shares(tmp);
 	}
 
-	if (affine_sd && wake_affine(affine_sd, p, sync))
+	if (affine_sd && wake_affine(affine_sd, p, cpu, sync))
 		return cpu;
 
 	while (sd) {

Quoted 11 lines · Lin Ming

commit a03ecf08d7bbdd979d81163ea13d194fe21ad339

Author: Lin Ming <[email protected]>
Date:   Mon Jan 4 14:14:50 2010 +0800

sched: Pass affine target cpu into wake_affine

Since commit a1f84a3(sched: Check for an idle shared cache in select_task_rq_fair()), the affine target maybe adjusted to any idle cpu in cache sharing domains instead of current cpu.
But wake_affine still use current cpu to calculate load which is wrong.

Quoted 3 lines

This patch passes affine cpu into wake_affine.

Signed-off-by Lin Ming <[email protected]>

Mike,

Any comment of this patch?

Quoted 16 lines · Lin Ming

Quoted 14 lines · Lin Ming

commit a03ecf08d7bbdd979d81163ea13d194fe21ad339

Author: Lin Ming <[email protected]>
Date:   Mon Jan 4 14:14:50 2010 +0800

sched: Pass affine target cpu into wake_affine

Since commit a1f84a3(sched: Check for an idle shared cache in select_task_rq_fair()), the affine target maybe adjusted to any idle cpu in cache sharing domains instead of current cpu.
But wake_affine still use current cpu to calculate load which is wrong.

This patch passes affine cpu into wake_affine.

Signed-off-by Lin Ming <[email protected]>

Quoted 3 lines

Mike,

Any comment of this patch?

The patch definitely looks like the right thing to do, but when I tried this, it didn't work out well. Since I can't seem to recall precise details, I'll let my box either remind me or give it's ack.

Quoted 17 lines · Mike Galbraith

Quoted 16 lines · Lin Ming

Quoted 14 lines · Lin Ming

commit a03ecf08d7bbdd979d81163ea13d194fe21ad339

Author: Lin Ming <[email protected]>
Date:   Mon Jan 4 14:14:50 2010 +0800

sched: Pass affine target cpu into wake_affine

Since commit a1f84a3(sched: Check for an idle shared cache in select_task_rq_fair()), the affine target maybe adjusted to any idle cpu in cache sharing domains instead of current cpu.
But wake_affine still use current cpu to calculate load which is wrong.

This patch passes affine cpu into wake_affine.

Signed-off-by Lin Ming <[email protected]>

Quoted 7 lines

Quoted 3 lines

Mike,

Any comment of this patch?

The patch definitely looks like the right thing to do, but when I tried this, it didn't work out well. Since I can't seem to recall precise details, I'll let my box either remind me or give it's ack.

Unfortunately, box reminded me. mysql+oltp peak throughput with nr_clients == nr_cpus

tip 37012.34
tip+ 33025.83

          .892

We really only want to check for shared cache on ramp-up and/or longish intermission. Once there's enough work to go around, interleaving is a big problem for these synchronous tasks. Doing the silly thing gets us the ramp-up gain without too much pain, though there is definitely pain for very fast switchers.

Looking always costs you a cache miss, not looking costs you throughput on ramp/intermission. Damned if you do, damned if you don't.

Quoted 19 lines · Mike Galbraith

Quoted 18 lines · Lin Ming

Quoted 6 lines · Lin Ming

commit a03ecf08d7bbdd979d81163ea13d194fe21ad339

Author: Lin Ming <[email protected]>
Date:   Mon Jan 4 14:14:50 2010 +0800

sched: Pass affine target cpu into wake_affine

Quoted 8 lines

Since commit a1f84a3(sched: Check for an idle shared cache in select_task_rq_fair()), the affine target maybe adjusted to any idle cpu in cache sharing domains instead of current cpu.
But wake_affine still use current cpu to calculate load which is wrong.

This patch passes affine cpu into wake_affine.

Signed-off-by Lin Ming <[email protected]>

Mike,

Quoted 5 lines

Quoted 1 line

Any comment of this patch?

The patch definitely looks like the right thing to do, but when I tried this, it didn't work out well. Since I can't seem to recall precise details, I'll let my box either remind me or give it's ack.

Unfortunately, box reminded me. mysql+oltp peak throughput with nr_clients == nr_cpus

I tested on below 2 machines with the 2 patches both applied and the oltp(sysbench+mysql) data shows good.
Tigerton x86_64 machine: 16cpus(4P/4Cores), 40G mem
IA64 machine: 32cpus(4P/4Cores/HT), 16G mem

Compared with upstream 2.6.33-rc2, IA64 improves ~15% and Tigerton improves ~3%.

Quoted 22 lines · Mike Galbraith

Quoted 21 lines · Mike Galbraith

Quoted 19 lines · Lin Ming

Quoted 6 lines · Lin Ming

commit a03ecf08d7bbdd979d81163ea13d194fe21ad339

Author: Lin Ming <[email protected]>
Date:   Mon Jan 4 14:14:50 2010 +0800

sched: Pass affine target cpu into wake_affine

Quoted 8 lines

Since commit a1f84a3(sched: Check for an idle shared cache in select_task_rq_fair()), the affine target maybe adjusted to any idle cpu in cache sharing domains instead of current cpu.
But wake_affine still use current cpu to calculate load which is wrong.

This patch passes affine cpu into wake_affine.

Signed-off-by Lin Ming <[email protected]>

Mike,

Any comment of this patch?

Quoted 6 lines

Quoted 3 lines

The patch definitely looks like the right thing to do, but when I tried this, it didn't work out well. Since I can't seem to recall precise details, I'll let my box either remind me or give it's ack.

Unfortunately, box reminded me. mysql+oltp peak throughput with nr_clients == nr_cpus

Did you test with your vmark regression fix patch also applied?

Quoted 22 lines · Mike Galbraith

Quoted 21 lines · Mike Galbraith

Quoted 19 lines · Lin Ming

Quoted 6 lines · Lin Ming

commit a03ecf08d7bbdd979d81163ea13d194fe21ad339

Author: Lin Ming <[email protected]>
Date:   Mon Jan 4 14:14:50 2010 +0800

sched: Pass affine target cpu into wake_affine

Quoted 8 lines

Since commit a1f84a3(sched: Check for an idle shared cache in select_task_rq_fair()), the affine target maybe adjusted to any idle cpu in cache sharing domains instead of current cpu.
But wake_affine still use current cpu to calculate load which is wrong.

This patch passes affine cpu into wake_affine.

Signed-off-by Lin Ming <[email protected]>

Mike,

Any comment of this patch?

Quoted 6 lines

Quoted 3 lines

The patch definitely looks like the right thing to do, but when I tried this, it didn't work out well. Since I can't seem to recall precise details, I'll let my box either remind me or give it's ack.

Unfortunately, box reminded me. mysql+oltp peak throughput with nr_clients == nr_cpus

Did you test with your vmark regression fix patch also applied?

Quoted 27 lines · Lin Ming

Quoted 26 lines · Mike Galbraith

Quoted 24 lines · Mike Galbraith

Quoted 12 lines · Lin Ming

Quoted 11 lines · Lin Ming

commit a03ecf08d7bbdd979d81163ea13d194fe21ad339

Author: Lin Ming <[email protected]>
Date:   Mon Jan 4 14:14:50 2010 +0800

sched: Pass affine target cpu into wake_affine

Since commit a1f84a3(sched: Check for an idle shared cache in select_task_rq_fair()), the affine target maybe adjusted to any idle cpu in cache sharing domains instead of current cpu.
But wake_affine still use current cpu to calculate load which is wrong.

Quoted 7 lines

Quoted 3 lines

This patch passes affine cpu into wake_affine.

Signed-off-by Lin Ming <[email protected]>

Mike,

Any comment of this patch?

The patch definitely looks like the right thing to do, but when I tried this, it didn't work out well. Since I can't seem to recall precise details, I'll let my box either remind me or give it's ack.

Quoted 4 lines

Quoted 2 lines

Unfortunately, box reminded me. mysql+oltp peak throughput with nr_clients == nr_cpus

Did you test with your vmark regression fix patch also applied?

Below is a complete retest. Mind testing my hacklet? I bet a nickle it'll work at least as well as yours on your beefy boxen.

Quoted 2 lines · Mike Galbraith

Below is a complete retest. Mind testing my hacklet? I bet a nickle it'll work at least as well as yours on your beefy boxen.

I tested your hacklet on below 2 machines as before.

Tigerton x86_64 machine: 16cpus(4P/4Cores), 40G mem IA64 machine: 32cpus(4P/4Cores/HT), 16G mem

Test1: vmark regression fix patch + pass affine target
Test2: this hacklet

Compared with upstream 2.6.33-rc2,
Test1: Tigerton +3%, IA64 +15%
Test2: Tigerton +3%, IA64 +10%

The test2 also improves on IA64, although not as good as test1.