Skip to contentSkip to conversation

sched,numa: always try to migrate to preferred node at task_numa_placement time

PATCH In review Rik van Riel 3 participants 5 messages Jun 4, 2014 – Jun 19, 2014 Scheduler

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 sched,numa: always try to migrate to preferred node at task_numa_placement time

It is possible that at task_numa_placement time, the task's numa_preferred_nid does not change, but the task is not actually running on the preferred node at the time.

In that case, we still want to attempt migration to the preferred node.

---
 kernel/sched/fair.c | 12 +++++++-----
 1 file changed, 7 insertions(+), 5 deletions(-)
kernel/sched/fair.c +7 -5
index 0fdb96d..708c6a9 100644
@@ -1575,11 +1575,13 @@ static void task_numa_placement(struct task_struct *p)
 		spin_unlock_irq(group_lock);
 	}
 
-	/* Preferred node as the node with the most faults */
-	if (max_faults && max_nid != p->numa_preferred_nid) {
-		/* Update the preferred nid and migrate task if possible */
-		sched_setnuma(p, max_nid);
-		numa_migrate_preferred(p);
+	if (max_faults) {
+		/* Set the new preferred node */
+		if (max_nid != p->numa_preferred_nid)
+			sched_setnuma(p, max_nid);
+
+		if (task_node(p) != p->numa_preferred_nid)
+			numa_migrate_preferred(p);
 	}
 }
 
maintainer
#1 lore raw
Quoted 6 lines · Rik van Riel

It is possible that at task_numa_placement time, the task's numa_preferred_nid does not change, but the task is not actually running on the preferred node at the time.

In that case, we still want to attempt migration to the preferred node.

So we have that numa_migrate_retry which was supposed to keep kicking the task until it got where it needed to go.

But now you continuously kick from task_numa_placement().

Clearly the retry thing didn't work, what happened? We got to the preferred nid, disabled the retry and got moved away again?

Do we want to remove the retry logic in favour of this more aggressive form?

Quoted 11 lines
kernel/sched/fair.c +7 -0
@@ -1575,11 +1575,13 @@ static void task_numa_placement(struct task_struct *p)
 
+	if (max_faults) {
+		/* Set the new preferred node */
+		if (max_nid != p->numa_preferred_nid)
+			sched_setnuma(p, max_nid);
+
+		if (task_node(p) != p->numa_preferred_nid)
+			numa_migrate_preferred(p);
 	}
 
#2 lore raw
Quoted 10 lines · Peter Zijlstra
Quoted 6 lines · Rik van Riel

It is possible that at task_numa_placement time, the task's numa_preferred_nid does not change, but the task is not actually running on the preferred node at the time.

In that case, we still want to attempt migration to the preferred node.

So we have that numa_migrate_retry which was supposed to keep kicking the task until it got where it needed to go.

It does, and it appears to work.

Quoted 1 line

But now you continuously kick from task_numa_placement().

No, we only kick from task_numa_placement() if the task is not already running on its preferred nid.

Quoted 5 lines

Clearly the retry thing didn't work, what happened? We got to the preferred nid, disabled the retry and got moved away again?

Do we want to remove the retry logic in favour of this more aggressive form?

I think we want both. When we have fresh statistics, and we discover that we are not running on our preferred nid, is there any reason not to relocate to a better node?

Moving a task to another node is cheap, and moving it sooner means we can end up avoiding migrating memory around twice.

Quoted 13 lines
Quoted 11 lines
kernel/sched/fair.c +7 -0
@@ -1575,11 +1575,13 @@ static void task_numa_placement(struct task_struct *p)
 
+	if (max_faults) {
+		/* Set the new preferred node */
+		if (max_nid != p->numa_preferred_nid)
+			sched_setnuma(p, max_nid);
+
+		if (task_node(p) != p->numa_preferred_nid)
+			numa_migrate_preferred(p);
 	}
 
Signature · 1 line
All rights reversed
maintainer
#3 lore raw
Quoted 11 lines · Rik van Riel
Quoted 5 lines · Peter Zijlstra

Clearly the retry thing didn't work, what happened? We got to the preferred nid, disabled the retry and got moved away again?

Do we want to remove the retry logic in favour of this more aggressive form?

I think we want both. When we have fresh statistics, and we discover that we are not running on our preferred nid, is there any reason not to relocate to a better node?

I got confused on the exact details, I thought we ended up calling task_numa_placement() far often than we actually do (well, we do call it as often, but bail early).

Yes this makes sense.

#4 lore raw

[tip:sched/core] sched/numa: Always try to migrate to preferred node at task_numa_placement() time

In reply to Rik van Riel
sched/numa: Always try to migrate to preferred node at task_numa_placement() time
Commit-ID:  bb97fc31647539f1f102eed646a95e200160a150
Gitweb:     http://git.kernel.org/tip/bb97fc31647539f1f102eed646a95e200160a150
Author:     Rik van Riel <[email protected]>
AuthorDate: Wed, 4 Jun 2014 16:33:15 -0400
Committer:  Ingo Molnar <[email protected]>
CommitDate: Wed, 18 Jun 2014 18:29:58 +0200

sched/numa: Always try to migrate to preferred node at task_numa_placement() time

It is possible that at task_numa_placement() time, the task's numa_preferred_nid does not change, but the task is not actually running on the preferred node at the time.

In that case, we still want to attempt migration to the preferred node.

---
 kernel/sched/fair.c | 12 +++++++-----
 1 file changed, 7 insertions(+), 5 deletions(-)
kernel/sched/fair.c +7 -5
index 8fbb011..3fa3e18 100644
@@ -1613,11 +1613,13 @@ static void task_numa_placement(struct task_struct *p)
 		spin_unlock_irq(group_lock);
 	}
 
-	/* Preferred node as the node with the most faults */
-	if (max_faults && max_nid != p->numa_preferred_nid) {
-		/* Update the preferred nid and migrate task if possible */
-		sched_setnuma(p, max_nid);
-		numa_migrate_preferred(p);
+	if (max_faults) {
+		/* Set the new preferred node */
+		if (max_nid != p->numa_preferred_nid)
+			sched_setnuma(p, max_nid);
+
+		if (task_node(p) != p->numa_preferred_nid)
+			numa_migrate_preferred(p);
 	}
 }