Allocate idle task for a CPU always on its local node

PATCH Applied Andi Kleen 4 participants 4 messages May 17, 2016 – May 18, 2016

Subsystems (3)

MM core Scheduler exec and binfmt

lore

Share & tools

lore

Andi Kleen

author

2016-05-17 20:59 UTC #0 lore raw

PATCH 1/1 Allocate idle task for a CPU always on its local node

From: Andi Kleen <[email protected]>

Linux pre-allocates the task structs of the idle tasks for all possible CPUs. This currently means they all end up on node 0. This also implies that the cache line of MWAIT, which is around the flags field in the task struct, are all located in node 0.

We see a noticeable performance improvement on Knights Landing CPUs when the cache lines used for MWAIT are located in the local nodes of the CPUs using them. I would expect this to give a (likely slight) improvement on other systems too.

The patch implements placing the idle task in the node of its CPUs, by passing the right target node to copy_process()

Signed-off-by Andi Kleen <[email protected]>

---
 kernel/fork.c | 15 +++++++++------
 1 file changed, 9 insertions(+), 6 deletions(-)

kernel/fork.c +9 -6

index d277e83..da449a9 100644
@@ -340,13 +340,14 @@ void set_task_stack_end_magic(struct task_struct *tsk)
 	*stackend = STACK_END_MAGIC;	/* for overflow detection */
 }
 
-static struct task_struct *dup_task_struct(struct task_struct *orig)
+static struct task_struct *dup_task_struct(struct task_struct *orig, int node)
 {
 	struct task_struct *tsk;
 	struct thread_info *ti;
-	int node = tsk_fork_get_node(orig);
 	int err;
 
+	if (node < 0)
+		node = tsk_fork_get_node(orig);
 	tsk = alloc_task_struct_node(node);
 	if (!tsk)
 		return NULL;

@@ -1256,7 +1257,8 @@ static struct task_struct *copy_process(unsigned long clone_flags,
 					int __user *child_tidptr,
 					struct pid *pid,
 					int trace,
-					unsigned long tls)
+					unsigned long tls,
+					int node)
 {
 	int retval;
 	struct task_struct *p;

@@ -1308,7 +1310,7 @@ static struct task_struct *copy_process(unsigned long clone_flags,
 		goto fork_out;
 
 	retval = -ENOMEM;
-	p = dup_task_struct(current);
+	p = dup_task_struct(current, node);
 	if (!p)
 		goto fork_out;

@@ -1684,7 +1686,8 @@ static inline void init_idle_pids(struct pid_link *links)
 struct task_struct *fork_idle(int cpu)
 {
 	struct task_struct *task;
-	task = copy_process(CLONE_VM, 0, 0, NULL, &init_struct_pid, 0, 0);
+	task = copy_process(CLONE_VM, 0, 0, NULL, &init_struct_pid, 0, 0,
+			    cpu_to_node(cpu));
 	if (!IS_ERR(task)) {
 		init_idle_pids(task->pids);
 		init_idle(task, cpu);

@@ -1729,7 +1732,7 @@ long _do_fork(unsigned long clone_flags,
 	}
 
 	p = copy_process(clone_flags, stack_start, stack_size,
-			 child_tidptr, NULL, trace, tls);
+			 child_tidptr, NULL, trace, tls, -1);
 	/*
 	 * Do this prior waking up the new thread - the thread pointer
 	 * might get invalid after that point, if the thread exits quickly.

Signature · 1 line

2.8.1

Andrew Morton

maintainer

2016-05-17 21:13 UTC #1 lore raw

Quoted 7 lines · On Tue, 17 May 2016 06:44:54 -0700 Andi Kleen <[email protected]> wrote:

From: Andi Kleen <[email protected]>
Linux pre-allocates the task structs of the idle tasks for all possible CPUs. This currently means they all end up on node 0. This also implies that the cache line of MWAIT, which is around the flags field in the task struct, are all located in node 0.

Quoted 8 lines

We see a noticeable performance improvement on Knights Landing CPUs when the cache lines used for MWAIT are located in the local nodes of the CPUs using them. I would expect this to give a (likely slight) improvement on other systems too.
The patch implements placing the idle task in the node of its CPUs, by passing the right target node to copy_process()

Looks nice.

This is nicer ;)

From: Andrew Morton <[email protected]>
Subject: allocate-idle-task-for-a-cpu-always-on-its-local-node-fix

use NUMA_NO_NODE, not a bare -1

Cc Andi Kleen <[email protected]>

Cc Thomas Gleixner <[email protected]>

Signed-off-by Andrew Morton <[email protected]>

---

 kernel/fork.c |    4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

kernel/fork.c renamed from kernel/fork.c~allocate-idle-task-for-a-cpu-always-on-its-local-node-fix +2 -2

diff -puN kernel/fork.c~allocate-idle-task-for-a-cpu-always-on-its-local-node-fix kernel/fork.c
@@ -346,7 +346,7 @@ static struct task_struct *dup_task_stru
 	struct thread_info *ti;
 	int err;
 
-	if (node < 0)
+	if (node == NUMA_NO_NODE)
 		node = tsk_fork_get_node(orig);
 	tsk = alloc_task_struct_node(node);
 	if (!tsk)

@@ -1754,7 +1754,7 @@ long _do_fork(unsigned long clone_flags,
 	}
 
 	p = copy_process(clone_flags, stack_start, stack_size,
-			 child_tidptr, NULL, trace, tls, -1);
+			 child_tidptr, NULL, trace, tls, NUMA_NO_NODE);
 	/*
 	 * Do this prior waking up the new thread - the thread pointer
 	 * might get invalid after that point, if the thread exits quickly.

Andi Kleen

2016-05-17 21:50 UTC #2 lore raw

Quoted 3 lines

Looks nice.
This is nicer ;)

Thanks. Looks indeed nicer.
-Andi

Thomas Gleixner

Acked-by on v1 1/1

2016-05-18 08:30 UTC #3 lore raw

In reply to Andi Kleen

Quoted 12 lines · Andi Kleen

From: Andi Kleen <[email protected]>
Linux pre-allocates the task structs of the idle tasks for all possible CPUs. This currently means they all end up on node 0. This also implies that the cache line of MWAIT, which is around the flags field in the task struct, are all located in node 0.
We see a noticeable performance improvement on Knights Landing CPUs when the cache lines used for MWAIT are located in the local nodes of the CPUs using them. I would expect this to give a (likely slight) improvement on other systems too.

Quoted 4 lines

The patch implements placing the idle task in the node of its CPUs, by passing the right target node to copy_process()
Signed-off-by Andi Kleen <[email protected]>

Acked-by Thomas Gleixner <[email protected]>