Memoryless nodes: Update memory policy and page migration

Online nodes now may have no memory. The checks and initialization must therefore be changed to no longer use the online functions.

This will correctly initialize the interleave on bootup to only target nodes with memory and will make sys_move_pages return an error when a page is to be moved to a memoryless node. Similarly we will get an error if MPOL_BIND and MPOL_INTERLEAVE is used on a memoryless node.

These are somewhat new semantics. So far one could specify memoryless nodes and we would maybe do the right thing and just ignore the node (or we'd do something strange like with MPOL_INTERLEAVE). If we want to allow the specification of memoryless nodes via memory policies then we need to keep checking for online nodes.

Signed-off-by Christoph Lameter <[email protected]>

Acked-by Nishanth Aravamudan <[email protected]>

---
 mm/mempolicy.c |   10 +++++-----
 mm/migrate.c   |    2 +-
 2 files changed, 6 insertions(+), 6 deletions(-)

Index: linux-2.6.22-rc6-mm1/mm/migrate.c
===================================================================

Quoted 7 lines

Index: linux-2.6.22-rc6-mm1/mm/migrate.c
===================================================================

mm/migrate.c +0 -0

@@ -963,7 +963,7 @@ asmlinkage long sys_move_pages(pid_t pid
 				goto out;
 

mm/migrate.c asmlinkage long sys_move_pages(pid_t pid · line 966

Quoted 3 lines

mm/migrate.c +1 -1

 			err = -ENODEV;
-			if (!node_online(node))
+			if (!node_memory(node))

if (!node_state(node, N_MEMORY))

?

mm/page_alloc.c line 56

Quoted 6 lines · Christoph Lameter

mm/page_alloc.c +5 -1

-EXPORT_SYMBOL(node_possible_map);
+nodemask_t node_states[NR_NODE_STATES] __read_mostly = {
+	[N_POSSIBLE] => NODE_MASK_ALL,
+	[N_ONLINE] =>{ { [0] = 1UL } }
+};
+EXPORT_SYMBOL(node_states);

Crap here too. I desperately need a vacation. Next week....

This fixes a problem I encountered testing Christoph's memoryless nodes series. Applies atop that series. Other than this, series holds up under what testing I've been able to do this week.

Memoryless Nodes: use "node_memory_map" for cpusets mems_allowed validation

cpusets try to ensure that any node added to a cpuset's mems_allowed is on-line and contains memory. The assumption was that online nodes contained memory. Thus, it is possible to add memoryless nodes to a cpuset and then add tasks to this cpuset. This results in continuous series of oom-kill and other console stack traces and apparent system hang.

Change cpusets to use node_states[N_MEMORY] [a.k.a. node_memory_map] in place of node_online_map when vetting memories. Return error if admin attempts to write a non-empty mems_allowed node mask containing only memoryless-nodes.

Signed-off-by Lee Schermerhorn <[email protected]>

This fixes a problem I encountered testing Christoph's memoryless nodes series. Applies atop that series. Other than this, series holds up under what testing I've been able to do this week.

Memoryless Nodes: use "node_memory_map" for cpusets mems_allowed validation

cpusets try to ensure that any node added to a cpuset's mems_allowed is on-line and contains memory. The assumption was that online nodes contained memory. Thus, it is possible to add memoryless nodes to a cpuset and then add tasks to this cpuset. This results in continuous series of oom-kill and other console stack traces and apparent system hang.

Change cpusets to use node_states[N_MEMORY] [a.k.a. node_memory_map] in place of node_online_map when vetting memories. Return error if admin attempts to write a non-empty mems_allowed node mask containing only memoryless-nodes.

Signed-off-by Lee Schermerhorn <[email protected]>

Or perhaps we should adjust cpusets to make it so that the mems_allowed member only includes nodes that are set in node_states[N_MEMORY]?

What do you think? Paul?

Quoted 4 lines

Or perhaps we should adjust cpusets to make it so that the mems_allowed member only includes nodes that are set in node_states[N_MEMORY]?

What do you think? Paul?

Do you mean the "mems_alloed member" of the task struct ?

That might make sense - changing task->mems_allowed to just include nodes with memory.

Someone would have to audit the entire kernel for uses of task->mems_allowed, to see if all uses would be ok with this change.

I'm on vacation this week and next, so won't be doing that work right now.

Quoted 13 lines · Lee Schermerhorn

This fixes a problem I encountered testing Christoph's memoryless nodes series. Applies atop that series. Other than this, series holds up under what testing I've been able to do this week.

Memoryless Nodes: use "node_memory_map" for cpusets mems_allowed validation

cpusets try to ensure that any node added to a cpuset's mems_allowed is on-line and contains memory. The assumption was that online nodes contained memory. Thus, it is possible to add memoryless nodes to a cpuset and then add tasks to this cpuset. This results in continuous series of oom-kill and other console stack traces and apparent system hang.

Quoted 6 lines

Change cpusets to use node_states[N_MEMORY] [a.k.a. node_memory_map] in place of node_online_map when vetting memories. Return error if admin attempts to write a non-empty mems_allowed node mask containing only memoryless-nodes.

Signed-off-by Lee Schermerhorn <[email protected]>

Lee, while looking at this change, I think it ends up fixing cpuset_mems_allowed() to return nodemasks that only include nodes in node_states[N_MEMORY]. However, cpuset_current_mems_allowed is a lockless macro which would still be broken. I think it would need to becom a static inline nodes_and() in the CPUSET case and a #define node_states[N_MEMORY] in the non-CPUSET case?

Or perhaps we should adjust cpusets to make it so that the mems_allowed member only includes nodes that are set in node_states[N_MEMORY]?

Quoted 28 lines · Nishanth Aravamudan

Quoted 13 lines · Lee Schermerhorn

This fixes a problem I encountered testing Christoph's memoryless nodes series. Applies atop that series. Other than this, series holds up under what testing I've been able to do this week.

Memoryless Nodes: use "node_memory_map" for cpusets mems_allowed validation

cpusets try to ensure that any node added to a cpuset's mems_allowed is on-line and contains memory. The assumption was that online nodes contained memory. Thus, it is possible to add memoryless nodes to a cpuset and then add tasks to this cpuset. This results in continuous series of oom-kill and other console stack traces and apparent system hang.

Quoted 6 lines

Change cpusets to use node_states[N_MEMORY] [a.k.a. node_memory_map] in place of node_online_map when vetting memories. Return error if admin attempts to write a non-empty mems_allowed node mask containing only memoryless-nodes.

Signed-off-by Lee Schermerhorn <[email protected]>

Lee, while looking at this change, I think it ends up fixing cpuset_mems_allowed() to return nodemasks that only include nodes in node_states[N_MEMORY]. However, cpuset_current_mems_allowed is a lockless macro which would still be broken. I think it would need to becom a static inline nodes_and() in the CPUSET case and a #define node_states[N_MEMORY] in the non-CPUSET case?

Quoted 2 lines

Or perhaps we should adjust cpusets to make it so that the mems_allowed member only includes nodes that are set in node_states[N_MEMORY]?

I thought that's what my patch to nodelist_parse() did. It ensures that current->mems_allowed is correct [contains at least one node with memory, and only nodes with memory] at the time it is installed, but doesn't consider memory hot plug and node off-lining. Is this [offline/hotplug] your point?

Seems like that is an issue that exists in the unpatched code as well--i.e., unlike cpuset_mems_allowed(), the lockless, "_current_" version does not vet current->mems_allowed against the nodes_online_mask. So, all valid nodes in current->mems_allowed could have been off-lined since the mask was installed. Am I reading this right?

Quoted 33 lines · Lee Schermerhorn

Quoted 28 lines · Nishanth Aravamudan

Quoted 13 lines · Lee Schermerhorn

This fixes a problem I encountered testing Christoph's memoryless nodes series. Applies atop that series. Other than this, series holds up under what testing I've been able to do this week.

Memoryless Nodes: use "node_memory_map" for cpusets mems_allowed validation

cpusets try to ensure that any node added to a cpuset's mems_allowed is on-line and contains memory. The assumption was that online nodes contained memory. Thus, it is possible to add memoryless nodes to a cpuset and then add tasks to this cpuset. This results in continuous series of oom-kill and other console stack traces and apparent system hang.

Quoted 6 lines

Change cpusets to use node_states[N_MEMORY] [a.k.a. node_memory_map] in place of node_online_map when vetting memories. Return error if admin attempts to write a non-empty mems_allowed node mask containing only memoryless-nodes.

Signed-off-by Lee Schermerhorn <[email protected]>

Lee, while looking at this change, I think it ends up fixing cpuset_mems_allowed() to return nodemasks that only include nodes in node_states[N_MEMORY]. However, cpuset_current_mems_allowed is a lockless macro which would still be broken. I think it would need to becom a static inline nodes_and() in the CPUSET case and a #define node_states[N_MEMORY] in the non-CPUSET case?

Quoted 2 lines

Or perhaps we should adjust cpusets to make it so that the mems_allowed member only includes nodes that are set in node_states[N_MEMORY]?

Quoted 5 lines

I thought that's what my patch to nodelist_parse() did. It ensures that current->mems_allowed is correct [contains at least one node with memory, and only nodes with memory] at the time it is installed, but doesn't consider memory hot plug and node off-lining. Is this [offline/hotplug] your point?

And everytime it is updated, right? (current->mems_allowed). My concern
is purely whether I can then directly use cpuset_current_mems_allowed in the interleave code for hugetlb.c and it will do the right thing. It will work, if the #define is changed for !CPUSETS and if your change guarantess current->mems_allowed is always consistent with node_states[N_MEMORY].

I think I simply was confused about the full impact of your changes, as I don't know cpusets that well. I'm going to try and test a memoryless node box I have at work w/ your change, though, and see what happens.

Quoted 7 lines

Seems like that is an issue that exists in the unpatched code as well--i.e., unlike cpuset_mems_allowed(), the lockless, "_current_" version does not vet current->mems_allowed against the nodes_online_mask. So, all valid nodes in current->mems_allowed could have been off-lined since the mask was installed. Am I reading this right?

True -- I honestly don't know. I doubt much of this code has been fully audited for full node unplug?

Quoted 40 lines · Nishanth Aravamudan

Quoted 33 lines · Lee Schermerhorn

Quoted 28 lines · Nishanth Aravamudan

Quoted 13 lines · Lee Schermerhorn

This fixes a problem I encountered testing Christoph's memoryless nodes series. Applies atop that series. Other than this, series holds up under what testing I've been able to do this week.

Memoryless Nodes: use "node_memory_map" for cpusets mems_allowed validation

cpusets try to ensure that any node added to a cpuset's mems_allowed is on-line and contains memory. The assumption was that online nodes contained memory. Thus, it is possible to add memoryless nodes to a cpuset and then add tasks to this cpuset. This results in continuous series of oom-kill and other console stack traces and apparent system hang.

Quoted 6 lines

Change cpusets to use node_states[N_MEMORY] [a.k.a. node_memory_map] in place of node_online_map when vetting memories. Return error if admin attempts to write a non-empty mems_allowed node mask containing only memoryless-nodes.

Signed-off-by Lee Schermerhorn <[email protected]>

Lee, while looking at this change, I think it ends up fixing cpuset_mems_allowed() to return nodemasks that only include nodes in node_states[N_MEMORY]. However, cpuset_current_mems_allowed is a lockless macro which would still be broken. I think it would need to becom a static inline nodes_and() in the CPUSET case and a #define node_states[N_MEMORY] in the non-CPUSET case?

Quoted 2 lines

Or perhaps we should adjust cpusets to make it so that the mems_allowed member only includes nodes that are set in node_states[N_MEMORY]?

Quoted 5 lines

I thought that's what my patch to nodelist_parse() did. It ensures that current->mems_allowed is correct [contains at least one node with memory, and only nodes with memory] at the time it is installed, but doesn't consider memory hot plug and node off-lining. Is this [offline/hotplug] your point?

Quoted 6 lines

And everytime it is updated, right? (current->mems_allowed). My concern
is purely whether I can then directly use cpuset_current_mems_allowed in the interleave code for hugetlb.c and it will do the right thing. It will work, if the #define is changed for !CPUSETS and if your change guarantess current->mems_allowed is always consistent with node_states[N_MEMORY].

Other than offlining/hot removal of memory, I think the only place that current->mems_allowed gets updated in in update_nodelist() [I wrote nodelist_parse() previously by mistake]. My patch to that function tries to ensure that current->mems_allowed always contains at least one node with memory.

If by "gets updated" you're referring to "cpuset_update_task_memory_state(), the latter calls "guarantee_online_mems()", which I also patched to use node_states[N_MEMORY] instead of "node_online_map". So, I think you can use current->mems_allowed in the hugetlb code. Maybe call "cpuset_update_task_memory_state()" before using it? However, I think that will have the effect of escaping the cpuset constraints if all of the nodes in the current task's mems_allowed have been offlined or hot removed since this mask was created/updated in update_nodelist().

Quoted 4 lines

I think I simply was confused about the full impact of your changes, as I don't know cpusets that well. I'm going to try and test a memoryless node box I have at work w/ your change, though, and see what happens.

FYI: I initially tried to test Christoph's memless nodes series with your rebased hugetlb patches, but the system appeared to hang. [Might be related to Ken Chen's recent hugetlb patch?]

I backed off to just Christoph's series and things seem to run OK. That's when I noticed that one could create a cpuset with just memoryless nodes and posted the subject patch. I'll get back to testing your patches on my memoryless nodes system "real soon now".

Meanwhile, as you've pointed out, I missed the "node_online_map" usage in the header and, I see, in the initialization of the top level cpuset in cpuset_init_smp(). I'm testing this now. I'll repost the patch with these fixes shortly.

For completeness, here's the numactl --hardware output [less the SLIT info] from my test platform [ia64] in it's current config:

Memoryless Nodes: use "node_memory_map" for cpusets - take 2

Against 2.6.22-rc6-mm1 atop Christoph Lameter's memoryless nodes series

take 2:
+ replaced node_online_map in cpuset_current_mems_allowed() with node_states[N_MEMORY]
+ replaced node_online_map in cpuset_init_smp() with node_states[N_MEMORY]

cpusets try to ensure that any node added to a cpuset's mems_allowed is on-line and contains memory. The assumption was that online nodes contained memory. Thus, it is possible to add memoryless nodes to a cpuset and then add tasks to this cpuset. This results in continuous series of oom-kill and apparent system hang.

Change cpusets to use node_states[N_MEMORY] [a.k.a. node_memory_map] in place of node_online_map when vetting memories. Return error if admin attempts to write a non-empty mems_allowed node mask containing only memoryless-nodes.

Quoted 18 lines · Lee Schermerhorn

Memoryless Nodes: use "node_memory_map" for cpusets - take 2

Against 2.6.22-rc6-mm1 atop Christoph Lameter's memoryless nodes series

take 2:
+ replaced node_online_map in cpuset_current_mems_allowed() with node_states[N_MEMORY]
+ replaced node_online_map in cpuset_init_smp() with node_states[N_MEMORY]

cpusets try to ensure that any node added to a cpuset's mems_allowed is on-line and contains memory. The assumption was that online nodes contained memory. Thus, it is possible to add memoryless nodes to a cpuset and then add tasks to this cpuset. This results in continuous series of oom-kill and apparent system hang.

Quoted 4 lines

Change cpusets to use node_states[N_MEMORY] [a.k.a. node_memory_map] in place of node_online_map when vetting memories. Return error if admin attempts to write a non-empty mems_allowed node mask containing only memoryless-nodes.

I think you still are missing a few comment changes (anything mentioning 'track'ing node_online_map will need to be changed, I think). Also, I don't see the necessary change in common_cpu_mem_hotplug_unplug() similar to cpuset_init_smp()'s change.

Quoted 24 lines · Nishanth Aravamudan

Quoted 18 lines · Lee Schermerhorn

Memoryless Nodes: use "node_memory_map" for cpusets - take 2

Against 2.6.22-rc6-mm1 atop Christoph Lameter's memoryless nodes series

take 2:
+ replaced node_online_map in cpuset_current_mems_allowed() with node_states[N_MEMORY]
+ replaced node_online_map in cpuset_init_smp() with node_states[N_MEMORY]

cpusets try to ensure that any node added to a cpuset's mems_allowed is on-line and contains memory. The assumption was that online nodes contained memory. Thus, it is possible to add memoryless nodes to a cpuset and then add tasks to this cpuset. This results in continuous series of oom-kill and apparent system hang.

Quoted 4 lines

Change cpusets to use node_states[N_MEMORY] [a.k.a. node_memory_map] in place of node_online_map when vetting memories. Return error if admin attempts to write a non-empty mems_allowed node mask containing only memoryless-nodes.

Quoted 4 lines

I think you still are missing a few comment changes (anything mentioning 'track'ing node_online_map will need to be changed, I think). Also, I don't see the necessary change in common_cpu_mem_hotplug_unplug() similar to cpuset_init_smp()'s change.

Sorry. Multitasking meltdown... Will fix.

Meanwhile:

I've tested your 3 patches atop Christoph's series [on 22-rc6-mm1], with and without my cpuset patch and I can't reproduce the hang I saw a couple of days ago :-(. I hate it when that happens! Perhaps some system daemon started up during the test that hung.

Memoryless Nodes: use "node_memory_map" for cpusets - take 3

Against 2.6.22-rc6-mm1 atop Christoph Lameter's memoryless nodes series

take 2:
+ replaced node_online_map in cpuset_current_mems_allowed() with node_states[N_MEMORY]
+ replaced node_online_map in cpuset_init_smp() with node_states[N_MEMORY]

take 3:
+ fix up comments and top level cpuset tracking of nodes with memory [instead of on-line nodes].
+ maybe I got them all this time?

cpusets try to ensure that any node added to a cpuset's mems_allowed is on-line and contains memory. The assumption was that online nodes contained memory. Thus, it is possible to add memoryless nodes to a cpuset and then add tasks to this cpuset. This results in continuous series of oom-kill and apparent system hang.

Change cpusets to use node_states[N_MEMORY] [a.k.a. node_memory_map] in place of node_online_map when vetting memories. Return error if admin attempts to write a non-empty mems_allowed node mask containing only memoryless-nodes.

Signed-off-by Lee Schermerhorn <[email protected]>

Memoryless Nodes: use "node_memory_map" for cpusets - take 3

Against 2.6.22-rc6-mm1 atop Christoph Lameter's memoryless nodes series

take 2:
+ replaced node_online_map in cpuset_current_mems_allowed() with node_states[N_MEMORY]
+ replaced node_online_map in cpuset_init_smp() with node_states[N_MEMORY]

take 3:
+ fix up comments and top level cpuset tracking of nodes with memory [instead of on-line nodes].
+ maybe I got them all this time?

Quoted 11 lines · Lee Schermerhorn

Memoryless Nodes: use "node_memory_map" for cpusets - take 3

Against 2.6.22-rc6-mm1 atop Christoph Lameter's memoryless nodes series

take 2:
+ replaced node_online_map in cpuset_current_mems_allowed() with node_states[N_MEMORY]
+ replaced node_online_map in cpuset_init_smp() with node_states[N_MEMORY]

Quoted 4 lines

take 3:
+ fix up comments and top level cpuset tracking of nodes with memory [instead of on-line nodes].
+ maybe I got them all this time?

My ack stands, but I believe Documentation/cpusets.txt will need updating too :)

Memoryless Nodes: use "node_memory_map" for cpusets - take 3

Against 2.6.22-rc6-mm1 atop Christoph Lameter's memoryless nodes series

take 2:
+ replaced node_online_map in cpuset_current_mems_allowed() with node_states[N_MEMORY]
+ replaced node_online_map in cpuset_init_smp() with node_states[N_MEMORY]

take 3:
+ fix up comments and top level cpuset tracking of nodes with memory [instead of on-line nodes].
+ maybe I got them all this time?

cpusets try to ensure that any node added to a cpuset's mems_allowed is on-line and contains memory. The assumption was that online nodes contained memory. Thus, it is possible to add memoryless nodes to a cpuset and then add tasks to this cpuset. This results in continuous series of oom-kill and apparent system hang.

Change cpusets to use node_states[N_MEMORY] [a.k.a. node_memory_map] in place of node_online_map when vetting memories. Return error if admin attempts to write a non-empty mems_allowed node mask containing only memoryless-nodes.

Signed-off-by Lee Schermerhorn <[email protected]>

 include/linux/cpuset.h |    2 -
 kernel/cpuset.c        |   51 +++++++++++++++++++++++++++++++------------------
 2 files changed, 34 insertions(+), 19 deletions(-)

Quoted 28 lines · Lee Schermerhorn

Memoryless Nodes: use "node_memory_map" for cpusets - take 3

Against 2.6.22-rc6-mm1 atop Christoph Lameter's memoryless nodes series

take 2:
+ replaced node_online_map in cpuset_current_mems_allowed() with node_states[N_MEMORY]
+ replaced node_online_map in cpuset_init_smp() with node_states[N_MEMORY]

take 3:
+ fix up comments and top level cpuset tracking of nodes with memory [instead of on-line nodes].
+ maybe I got them all this time?

cpusets try to ensure that any node added to a cpuset's mems_allowed is on-line and contains memory. The assumption was that online nodes contained memory. Thus, it is possible to add memoryless nodes to a cpuset and then add tasks to this cpuset. This results in continuous series of oom-kill and apparent system hang.

Change cpusets to use node_states[N_MEMORY] [a.k.a. node_memory_map] in place of node_online_map when vetting memories. Return error if admin attempts to write a non-empty mems_allowed node mask containing only memoryless-nodes.

Quoted 5 lines

Signed-off-by Lee Schermerhorn <[email protected]>

 include/linux/cpuset.h |    2 -
 kernel/cpuset.c        |   51 +++++++++++++++++++++++++++++++------------------
 2 files changed, 34 insertions(+), 19 deletions(-)

Small typo fix which prevents build with !CPUSETS.

---
FYI: I noticed that oldconfig on 2.6.23-rc1-mm1 with CPUSETS=y disables CPUSETS because of the introduction of CONFIG_CONTAINERS :(

Signed-off-by Nishanth Aravamudan <[email protected]>

include/linux/cpuset.h +0 -0

index f8f4f68..d01b1bc 100644

Suppress redundant "node with no memory" messages

Against 2.6.22-rc6-mm1 atop Christoph Lameter's memoryless node series.

get_pfn_range_for_nid() is called multiple times for each node at boot time. Each time, it will warn about nodes with no memory, resulting in boot messages like:

Node 0 active with no memory
Node 0 active with no memory
Node 0 active with no memory
Node 0 active with no memory
Node 0 active with no memory
Node 0 active with no memory
On node 0 totalpages: 0
Node 0 active with no memory
Node 0 active with no memory
DMA zone: 0 pages used for memmap
Node 0 active with no memory
Node 0 active with no memory
Normal zone: 0 pages used for memmap
Node 0 active with no memory
Node 0 active with no memory
Movable zone: 0 pages used for memmap

and so on for each memoryless node. Track [in init data] memoryless nodes that we've already reported to suppress the redundant messages.

OR, we could eliminate the message altogether? We do report zero totalpages. Sufficient?

Signed-off-by Lee Schermerhorn <[email protected]>

Simply switch all for_each_online_node to for_each_memory_node. That way SLUB only operates on nodes with memory. Any allocation attempt on a memoryless node will fall whereupon SLUB will fetch memory from a nearby node (depending on how memory policies and cpuset describe fallback).

This is as far as I got when a reject storm hit.

Review

Quoted 3 lines

(quoted diff) +1 -1

-	for_each_online_node(node)
+	for_each_node_state(node, N_MEMORY)
 		__kmem_cache_shrink(s, get_node(s, node), scratch);

I can find no sign of any __kmem_cache_shrink's anywhere.

Let's park all this until post-merge-window please. Generally, now is not a good time for me to be merging 2.6.24 stuff.

Quoted 7 lines · Andrew Morton

This is as far as I got when a reject storm hit.

mm/slub.c static unsigned long slab_objects(struct

Quoted 3 lines

mm/slub.c +1 -1

-	for_each_online_node(node)
+	for_each_node_state(node, N_MEMORY)
 		__kmem_cache_shrink(s, get_node(s, node), scratch);

I can find no sign of any __kmem_cache_shrink's anywhere.

Yup I expected slab defrag to be merged first before you get to this.

Quoted 2 lines

Let's park all this until post-merge-window please. Generally, now is not a good time for me to be merging 2.6.24 stuff.

For SGI this is not important at all since we have no memoryless nodes.

However, these fixes are important for other NUMA users. I think this needs to go into 2.6.23 for correctnesses sake. We may have some fun with it since the fixed up behavior of GFP_THISNODE may expose additional problems in how subsystems handle memoryless nodes (and I do not have such a system). There are also patches against hugetlb that use this functionality here.

Necessary for asymmetric NUMA configs to work right.

Here is the patch rediffed before slab defrag.

Memoryless nodes: SLUB support

Simply switch all for_each_online_node to for_each_memory_node. That way SLUB only operates on nodes with memory. Any allocation attempt on a memoryless node will fall whereupon SLUB will fetch memory from a nearby node (depending on how memory policies and cpuset describe fallback).

This description is out of date. There is no for_each_memory_node() any more, I think you meant for_each_node_state().

Changes V2->V3:
- Refresh patches (sigh)
- Add comments suggested by Kamezawa Hiroyuki
- Add signoff by Jes Sorensen

Quoted 4 lines · Christoph Lameter

Changes V2->V3:
- Refresh patches (sigh)
- Add comments suggested by Kamezawa Hiroyuki
- Add signoff by Jes Sorensen

Christoph, would it be possible to get the current patches up on kernel.org in your people-space? That way I know I have the current versions of these, including any fixlets that come by?

Quoted 6 lines · Nishanth Aravamudan

Quoted 4 lines · Christoph Lameter

Changes V2->V3:
- Refresh patches (sigh)
- Add comments suggested by Kamezawa Hiroyuki
- Add signoff by Jes Sorensen

Quoted 3 lines

Christoph, would it be possible to get the current patches up on kernel.org in your people-space? That way I know I have the current versions of these, including any fixlets that come by?

Lee: Would you repost the patches after testing them and fixing them up?

You probably have somewhere to publish them? I will be on vacation next week (and yes I will leave my laptop at home, somehow I have to get back my sanity).

Quoted 6 lines · Nishanth Aravamudan

Quoted 4 lines · Christoph Lameter

Changes V2->V3:
- Refresh patches (sigh)
- Add comments suggested by Kamezawa Hiroyuki
- Add signoff by Jes Sorensen

Quoted 3 lines

Christoph, would it be possible to get the current patches up on kernel.org in your people-space? That way I know I have the current versions of these, including any fixlets that come by?

Lee: Would you repost the patches after testing them and fixing them up?

You probably have somewhere to publish them? I will be on vacation next week (and yes I will leave my laptop at home, somehow I have to get back my sanity).

I'm up to my eyeballs right now, setting up a large system for testing VM scalability with Oracle. I hope to have time early next week to test your patches. In a mail exchange between you and Andrew, you mentioned that your memoryless-node patches are atop your slab defrag? Shall I test them that way? Or try to rebase against the then current -mm tree?

I.e., what's the probability that the slab defrag patches make it into -mm before the memoryless node patches?

Quoted 3 lines

You probably have somewhere to publish them? I will be on vacation next week (and yes I will leave my laptop at home, somehow I have to get back my sanity).

You mean in addition to posting? I can stick a copy on my free.linux.hp.com http site.

Quoted 5 lines · Lee Schermerhorn

I'm up to my eyeballs right now, setting up a large system for testing VM scalability with Oracle. I hope to have time early next week to test your patches. In a mail exchange between you and Andrew, you mentioned that your memoryless-node patches are atop your slab defrag? Shall I test them that way? Or try to rebase against the then current -mm tree?

You can skip the slab defrag. I posted a rediffed patch in my response to Andrew. Use that one.

Next week I'll reconfig a platform fully interleaved which will result in all of the real nodes appearing memoryless and do more testing.

Have a nice vacation.

Christoph:

Had a chance to build/boot the latest series with the updated #7, ...

Quite a few offsets and one reject in #7, but easy to resolve.

Boots OK. Quick test of hugetlb allocation on my platform shows the old behavior with huge pages doubling up on the node that the "memoryless" one falls back on. Guess this is expected until we get Nish's patch atop this one.

Next week I'll reconfig a platform fully interleaved which will result in all of the real nodes appearing memoryless and do more testing.

Have a nice vacation.

Nish:

Shall I try to rebase your patches atop Christoph's in my tree?

The last ones I have are from 19jul:

01-fix-hugetlb-pool-allocation-with-memoryless-nodes

02-hugetlb-numafy-several-functions 03-add-per-node-nr_hugepages-sysfs-attribute

Do you have more recent ones?

Quoted 1 line · Christoph Lameter

On Tue, 24 Jul 2007, Lee Schermerhorn wrote:

<snip>

Quoted 10 lines

Quoted 7 lines

I have tested your series, with Nish's and my patches in a memory-constrained config with all of the cpus on memoryless nodes and all of the memory in a cpu-less pseudo-node. Seems to hold up fairly well under stress. I did see a hang on Monday--test hung, very little free memory, pdflush just trickling out pages--but haven't been able to reproduce it. Don't know what happened.

Hmm... Not good. Was that with a pre rc1 release? I got a hang here on a simulator that seems to be related to high res timers.

It was on 22-rc6-mm1.

Quoted 8 lines

Quoted 7 lines

I haven't had a chance to poke at it with memtoy to see how the interleave and hugepages work. But, most folks don't use that, so I think it's appropriate for -mm.

How should we proceed? Shall I Ack the patches, mentioning the testing I've done and recommend inclusion in -mm?

Quoted 4 lines

Could you post the patchset with your acks or signoffs if you have made changes? Address them to Andrew, cc me and I will support merging of what you got. Note though that I think were are at the beginning of dealing with nodeless and per node memory use.

I'm rebasing to 23-rc1-mm1 right now. Will do a quick test and repost.

2) fails to boot with:

Unable to handle kernel paging request at virtual address a000400002000020
swapper[0]: Oops 11003706212352 [1]
Modules linked in:

Pid: 0, CPU 0, comm: swapper
psr : 00001210084a2010 ifs : 8000000000000995 ip : [<a0000001008a4df1>] Not tainted
ip is at memmap_init_zone+0x271/0x2a0

... then hard hang.

I tried to deselect SPARSEMEM_VMEMMAP. Kconfig's "def_bool=y" wouldn't let me :-(. After hacking the Kconfig and mm/sparse.c to allow that, boot hangs with no error messages shortly after "Built N zonelists..." message.

Backed off to DISCONTIGMEM+VIRTUAL_MEMORY_MAP, and saw same hang as with (SPARSMEM && !SPARSEMEM_VMEMMAP).

I should mention that I have my test system in the "fully interleaved" configuration for testing the memoryless node patches. This means that nodes 0-3 [the real nodes with the cpus attached] have no memory. All memory resides in a cpu-less pseudo-node. I'm wondering if SPARSEMEM_VMEMMAP can handle this? 22-rc6-mm1 booted OK on this config w/ SPARSEMEM_EXTREME.

I get a similar hang here and see the system looping in softirq / hrtimer code.

(ccing Andy who did the work on the config stuff)

Quoted 4 lines · Lee Schermerhorn

I tried to deselect SPARSEMEM_VMEMMAP. Kconfig's "def_bool=y" wouldn't let me :-(. After hacking the Kconfig and mm/sparse.c to allow that, boot hangs with no error messages shortly after "Built N zonelists..." message.

I get a similar hang here and see the system looping in softirq / hrtimer code.

Quoted 2 lines

Backed off to DISCONTIGMEM+VIRTUAL_MEMORY_MAP, and saw same hang as with (SPARSMEM && !SPARSEMEM_VMEMMAP).

So its not related to SPARSE VMEMMAP? General VMEMMAP issue on IA64?

Quoted 7 lines

I should mention that I have my test system in the "fully interleaved" configuration for testing the memoryless node patches. This means that nodes 0-3 [the real nodes with the cpus attached] have no memory. All memory resides in a cpu-less pseudo-node. I'm wondering if SPARSEMEM_VMEMMAP can handle this? 22-rc6-mm1 booted OK on this config w/ SPARSEMEM_EXTREME.

The vmemmap page table blocks get allocated on the nodes where there is actual mmemory but sparse.c may not have been updated to only look for memory on nodes that have memory. If it looks for online nodes then we may have an issue there. Andy?

Were you able to run discontig/vmemmap in the past with this configuration?

Quoted 15 lines · Christoph Lameter

(ccing Andy who did the work on the config stuff)

Quoted 4 lines · Lee Schermerhorn

I tried to deselect SPARSEMEM_VMEMMAP. Kconfig's "def_bool=y" wouldn't let me :-(. After hacking the Kconfig and mm/sparse.c to allow that, boot hangs with no error messages shortly after "Built N zonelists..." message.

I get a similar hang here and see the system looping in softirq / hrtimer code.

Quoted 2 lines

Backed off to DISCONTIGMEM+VIRTUAL_MEMORY_MAP, and saw same hang as with (SPARSMEM && !SPARSEMEM_VMEMMAP).

Quoted 1 line

So its not related to SPARSE VMEMMAP? General VMEMMAP issue on IA64?

This hang is different from the one I see with SPARSE VMEMMAP -- no "Unable to handle kernel paging request..." message. Just hangs after "Built N zonelists..." and some message about "color" that I didn't capture. Next time [:-(]...

Quoted 16 lines · Lee Schermerhorn

Quoted 15 lines · Christoph Lameter

(ccing Andy who did the work on the config stuff)

Quoted 4 lines · Lee Schermerhorn

I tried to deselect SPARSEMEM_VMEMMAP. Kconfig's "def_bool=y" wouldn't let me :-(. After hacking the Kconfig and mm/sparse.c to allow that, boot hangs with no error messages shortly after "Built N zonelists..." message.

I get a similar hang here and see the system looping in softirq / hrtimer code.

Quoted 2 lines

Backed off to DISCONTIGMEM+VIRTUAL_MEMORY_MAP, and saw same hang as with (SPARSMEM && !SPARSEMEM_VMEMMAP).

Quoted 6 lines

Quoted 1 line

So its not related to SPARSE VMEMMAP? General VMEMMAP issue on IA64?

This hang is different from the one I see with SPARSE VMEMMAP -- no "Unable to handle kernel paging request..." message. Just hangs after "Built N zonelists..." and some message about "color" that I didn't capture. Next time [:-(]...

The "color" message was actually:

Console: colour dummy device 80x25

So, now I'm wondering if I'm hitting the "Regression in serial console..." issue, and the system was actually booting--I just didn't see any output. If so, the "Unable to handle kernel paging request..." hang might well be a problem with SPARSEMEM_VMEMMAP...

Quoted 28 lines · On Thu, 26 Jul 2007 09:53:27 -0400 Lee Schermerhorn <[email protected]>…

Quoted 16 lines · Lee Schermerhorn

Quoted 15 lines · Christoph Lameter

(ccing Andy who did the work on the config stuff)

Quoted 4 lines · Lee Schermerhorn

I tried to deselect SPARSEMEM_VMEMMAP. Kconfig's "def_bool=y" wouldn't let me :-(. After hacking the Kconfig and mm/sparse.c to allow that, boot hangs with no error messages shortly after "Built N zonelists..." message.

I get a similar hang here and see the system looping in softirq / hrtimer code.

Quoted 2 lines

Backed off to DISCONTIGMEM+VIRTUAL_MEMORY_MAP, and saw same hang as with (SPARSMEM && !SPARSEMEM_VMEMMAP).

Quoted 6 lines

Quoted 1 line

So its not related to SPARSE VMEMMAP? General VMEMMAP issue on IA64?

This hang is different from the one I see with SPARSE VMEMMAP -- no "Unable to handle kernel paging request..." message. Just hangs after "Built N zonelists..." and some message about "color" that I didn't capture. Next time [:-(]...

The "color" message was actually:

Console: colour dummy device 80x25

Quoted 5 lines

So, now I'm wondering if I'm hitting the "Regression in serial console..." issue, and the system was actually booting--I just didn't see any output. If so, the "Unable to handle kernel paging request..." hang might well be a problem with SPARSEMEM_VMEMMAP...

About SPARSEMEM_VMEMMAP try this:
http://lkml.org/lkml/2007/7/26/161

Quoted 16 lines · Lee Schermerhorn

Quoted 15 lines · Christoph Lameter

(ccing Andy who did the work on the config stuff)

Quoted 4 lines · Lee Schermerhorn

I tried to deselect SPARSEMEM_VMEMMAP. Kconfig's "def_bool=y" wouldn't let me :-(. After hacking the Kconfig and mm/sparse.c to allow that, boot hangs with no error messages shortly after "Built N zonelists..." message.

I get a similar hang here and see the system looping in softirq / hrtimer code.

Quoted 2 lines

Backed off to DISCONTIGMEM+VIRTUAL_MEMORY_MAP, and saw same hang as with (SPARSMEM && !SPARSEMEM_VMEMMAP).

Quoted 6 lines

Quoted 1 line

So its not related to SPARSE VMEMMAP? General VMEMMAP issue on IA64?

This hang is different from the one I see with SPARSE VMEMMAP -- no "Unable to handle kernel paging request..." message. Just hangs after "Built N zonelists..." and some message about "color" that I didn't capture. Next time [:-(]...

The "color" message was actually:

Console: colour dummy device 80x25

So, now I'm wondering if I'm hitting the "Regression in serial console..." issue, and the system was actually booting--I just didn't see any output. If so, the "Unable to handle kernel paging request..." hang might well be a problem with SPARSEMEM_VMEMMAP...