Limit initial tasks' and top level cpuset's mems_allowed to nodes with memory

Against: 2.6.20-rc3-mmotm-090428-1631

Since cpusetmm-update-tasks-mems_allowed-in-time.patch removed the call outs to cpuset_update_task_memory_state(), tasks in the top cpuset don't get their mems_allowed updated to just nodes with memory. cpuset_init()initializes the top cpuset's mems_allowed with nodes_setall() and cpuset_init_current_mems_allowed() and kernel_init() initialize the kernel initialization tasks' mems_allowed to all possible nodes. Tasks in the top cpuset that inherit the init task's mems_allowed without modification will have all possible nodes set. This can be seen by examining the Mems_allowed field in /proc/<pid>/status in such a task.

"numactl --interleave=all" also initializes the interleave node mask to all ones, depending on the masking with mems_allowed to eliminate non-existent nodes and nodes without memory. As this was not happening, the interleave policy was attempting to dereference non-existent nodes.

This patch modifies the nodes_setall() calls in two cpuset init functions and the initialization of task #1's mems_allowed to use node_states[N_HIGH_MEMORY]. This mask has been initialized to contain only existing nodes with memory by the time the respective init functions are called.

This fixes the bogus pointer deref [Nat Consumption fault on ia64] reported in:

	[BUG] 2.6.30-rc3-mmotm-090428-1814 -- bogus pointer deref

[The time--1814--was incorrect in that subject line, but the date was correct.]

Signed-off-by Lee Schermerhorn <[email protected]>

 init/main.c     |    4 ++--
 kernel/cpuset.c |    4 ++--
 2 files changed, 4 insertions(+), 4 deletions(-)

Index: linux-2.6.30-rc3-mmotm-090428-1631/kernel/cpuset.c
===================================================================

kernel/cpuset.c +2 -2

@@ -1846,7 +1846,7 @@ int __init cpuset_init(void)
 		BUG();
 
 	cpumask_setall(top_cpuset.cpus_allowed);
-	nodes_setall(top_cpuset.mems_allowed);
+	top_cpuset.mems_allowed = node_states[N_HIGH_MEMORY];
 
 	fmeter_init(&top_cpuset.fmeter);
 	set_bit(CS_SCHED_LOAD_BALANCE, &top_cpuset.flags);

@@ -2118,7 +2118,7 @@ void cpuset_cpus_allowed_locked(struct t
 
 void cpuset_init_current_mems_allowed(void)
 {
-	nodes_setall(current->mems_allowed);
+	current->mems_allowed = node_states[N_HIGH_MEMORY];
 }
 
 /**

Index: linux-2.6.30-rc3-mmotm-090428-1631/init/main.c
===================================================================

init/main.c +0 -0

@@ -849,9 +849,9 @@ static int __init kernel_init(void * unu
 	lock_kernel();

Quoted 17 lines · on 2009-5-4 11:06 Lee Schermerhorn wrote:

Against: 2.6.20-rc3-mmotm-090428-1631

Since cpusetmm-update-tasks-mems_allowed-in-time.patch removed the call outs to cpuset_update_task_memory_state(), tasks in the top cpuset don't get their mems_allowed updated to just nodes with memory. cpuset_init()initializes the top cpuset's mems_allowed with nodes_setall() and cpuset_init_current_mems_allowed() and kernel_init() initialize the kernel initialization tasks' mems_allowed to all possible nodes. Tasks in the top cpuset that inherit the init task's mems_allowed without modification will have all possible nodes set. This can be seen by examining the Mems_allowed field in /proc/<pid>/status in such a task.

"numactl --interleave=all" also initializes the interleave node mask to all ones, depending on the masking with mems_allowed to eliminate non-existent nodes and nodes without memory. As this was not happening, the interleave policy was attempting to dereference non-existent nodes.

Quoted 4 lines

This patch modifies the nodes_setall() calls in two cpuset init functions and the initialization of task #1's mems_allowed to use node_states[N_HIGH_MEMORY]. This mask has been initialized to contain only existing nodes with memory by the time the respective init functions are called.

You forget to modify the cpuset_attach(). This function will initialize the mems_allowed of the task which is being moved into the top cpuset by node_possible_map.

Beside that, if you use node_states[N_HIGH_MEMORY] to initialize the mems_allowed of the tasks in the top cpuset, you must update it when adding a node with memory into the system. So you also must modify cpuset_track_online_nodes().