sched/numa: More NUMA_PULL_BIAS fixes

Commit-ID:  bb527e7df7bd5f05dca74868f61b7a9a17ee54c1
Gitweb:     http://git.kernel.org/tip/bb527e7df7bd5f05dca74868f61b7a9a17ee54c1
Author:     Peter Zijlstra <[email protected]>
AuthorDate: Thu, 11 Oct 2012 13:06:22 +0200
Committer:  Ingo Molnar <[email protected]>
CommitDate: Mon, 15 Oct 2012 13:56:49 +0200

sched/numa: More NUMA_PULL_BIAS fixes

Consider the simple case where 2 runnable tasks, each with a preferred node (not the same), each running on the wrong node. Weight wise, this is a well balanced situation and thus the load-balancer will not find anything to do and is content.

NUMA_PULL_BIAS is the 'force' introduced to break this situation and promote task movement despite the weight balance so that we might find the situation where each task runs on its preferred node.

It was (again) observed that this force was too strong and created undue task movement. In particular, suppose the two runnable tasks each having the same preferred node, the weight balance will then move one of them to the other node. This results in one task running on the right node and one on the wrong node.

NUMA_PULL_BIAS would in this case try and move the off-node task back to its home-node in order to promote better node-preference. This is clearly sub-optimal since either we end up moving that same task back, or worse, we'll move the other task to the now empty node.

This results in endless undue task movement.

Solve this by limiting the NUMA_PULL_BIAS to only allow moving an offnode task to its home-node when the home-node has either another offnode task or an non-numa task to trade back for it.

---
 kernel/sched/core.c  |    1 +
 kernel/sched/fair.c  |   70 +++++++++++++++++++++++++++++++++----------------
 kernel/sched/sched.h |   11 +-------
 3 files changed, 49 insertions(+), 33 deletions(-)

index c386297..23ad8b9 100644
@@ -6930,6 +6930,7 @@ void __init sched_init(void)
 		INIT_LIST_HEAD(&rq->cfs_tasks);
 #ifdef CONFIG_SCHED_NUMA
 		INIT_LIST_HEAD(&rq->offnode_tasks);
+		rq->onnode_running = 0;
 		rq->offnode_running = 0;
 		rq->offnode_weight = 0;
 #endif

index 3060136..e68cef1 100644
@@ -420,6 +420,7 @@ struct rq {
 	struct list_head cfs_tasks;
 
 #ifdef CONFIG_SCHED_NUMA
+	unsigned long    onnode_running;
 	unsigned long    offnode_running;
 	unsigned long	 offnode_weight;
 	struct list_head offnode_tasks;

@@ -494,11 +495,6 @@ DECLARE_PER_CPU(struct rq, runqueues);
 #define raw_rq()		(&__raw_get_cpu_var(runqueues))
 
 #ifdef CONFIG_SCHED_NUMA
-static inline bool offnode_task(struct task_struct *t)
-{
-	return t->node != -1 && t->node != cpu_to_node(task_cpu(t));
-}
-
 static inline struct list_head *offnode_tasks(struct rq *rq)
 {
 	return &rq->offnode_tasks;

@@ -509,11 +505,6 @@ static inline void task_numa_free(struct task_struct *p)
 	kfree(p->numa_faults);
 }
 #else /* CONFIG_SCHED_NUMA */
-static inline bool offnode_task(struct task_struct *t)
-{
-	return false;
-}
-
 static inline struct list_head *offnode_tasks(struct rq *rq)
 {
 	return NULL;