Skip to contentSkip to conversation

prefer allocations from nodes w/o CPUs

RFC In review Jesse Barnes 3 participants 8 messages Nov 4, 2004

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

RFC 1/1 prefer allocations from nodes w/o CPUs

Small patch for the ia64 version of topology.h to make the zonelist building code prefer nodes w/o CPUs for allocations. Do any platforms aside from Altix have nodes with only memory? If not, then this will only affect sn2, otherwise I'm interested in hearing feedback.

Thanks,
Jesse

===== include/asm-ia64/topology.h 1.13 vs edited =====

include/asm-ia64/topology.h +4 -0
@@ -18,6 +18,10 @@
 #include <asm/smp.h>
 
 #ifdef CONFIG_NUMA
+
+/* Nodes w/o CPUs are preferred for memory allocations, see build_zonelists */
+#define PENALTY_FOR_NODE_WITH_CPUS 255
+
 /*
  * Returns the number of the node containing CPU 'cpu'
  */
#1 lore raw
Quoted 4 lines · Jesse Barnes

Small patch for the ia64 version of topology.h to make the zonelist building code prefer nodes w/o CPUs for allocations. Do any platforms aside from Altix have nodes with only memory? If not, then this will only affect sn2, otherwise I'm interested in hearing feedback.

Yes, hp sx1000 systems represent interleaved memory in a separate
node w/o cpus.

Alex

Signature · 1 line
Alex Williamson                             HP Linux & Open Source Lab
#2 lore raw
In reply to Jesse Barnes
Quoted 8 lines · Alex Williamson
Quoted 4 lines · Jesse Barnes

Small patch for the ia64 version of topology.h to make the zonelist building code prefer nodes w/o CPUs for allocations. Do any platforms aside from Altix have nodes with only memory? If not, then this will only affect sn2, otherwise I'm interested in hearing feedback.

Yes, hp sx1000 systems represent interleaved memory in a separate
node w/o cpus.

So does this patch make sense for those platforms too? Or do we need something fancier? It'll make nodes w/o CPUs come before nodes with CPUs in the zonelist (after the local node of course), even if the nodes with CPUs are closer.

Jesse

#3 lore raw
In reply to Jesse Barnes
Quoted 7 lines · Jesse Barnes
Quoted 6 lines · Alex Williamson
Quoted 4 lines · Jesse Barnes

Small patch for the ia64 version of topology.h to make the zonelist building code prefer nodes w/o CPUs for allocations. Do any platforms aside from Altix have nodes with only memory? If not, then this will only affect sn2, otherwise I'm interested in hearing feedback.

Quoted 7 lines
Quoted 2 lines

Yes, hp sx1000 systems represent interleaved memory in a separate
node w/o cpus.

So does this patch make sense for those platforms too? Or do we need something fancier? It'll make nodes w/o CPUs come before nodes with CPUs in the zonelist (after the local node of course), even if the nodes with CPUs are closer.

That makes sense for us. The interleaved memory should be somewhat
faster than truly remote memory. Depending on the box configuration, the interleaved memory node could be quite small and we'll run out quickly, but better to make what use we can from it. Thanks,

Alex

Signature · 1 line
Alex Williamson                             HP Linux & Open Source Lab
#4 lore raw
In reply to Jesse Barnes
Quoted 8 lines · Alex Williamson
Quoted 4 lines · Jesse Barnes

Small patch for the ia64 version of topology.h to make the zonelist building code prefer nodes w/o CPUs for allocations. Do any platforms aside from Altix have nodes with only memory? If not, then this will only affect sn2, otherwise I'm interested in hearing feedback.

Yes, hp sx1000 systems represent interleaved memory in a separate
node w/o cpus.

So what's the ramifications of this patch? Does this mean that interleaved memory will now be preferably allocated over cell local memory on sx1000 systems?

Thanks,
Alex Tsariounov

#5 lore raw
In reply to Jesse Barnes
Quoted 7 lines · Alex Tsariounov
Quoted 6 lines · Alex Williamson
Quoted 4 lines · Jesse Barnes

Small patch for the ia64 version of topology.h to make the zonelist building code prefer nodes w/o CPUs for allocations. Do any platforms aside from Altix have nodes with only memory? If not, then this will only affect sn2, otherwise I'm interested in hearing feedback.

Quoted 6 lines
Quoted 2 lines

Yes, hp sx1000 systems represent interleaved memory in a separate
node w/o cpus.

So what's the ramifications of this patch? Does this mean that interleaved memory will now be preferably allocated over cell local memory on sx1000 systems?

Memory local to the node will come first, followed by the closest memory that doesn't have associated CPUs, followed by memory belonging to other CPUs.

Jesse

#6 lore raw
In reply to Jesse Barnes
Quoted 12 lines · Jesse Barnes
Quoted 11 lines · Alex Tsariounov
Quoted 9 lines · Alex Williamson
Quoted 5 lines · Jesse Barnes

Small patch for the ia64 version of topology.h to make the zonelist building code prefer nodes w/o CPUs for allocations. Do any platforms aside from Altix have nodes with only memory? If not, then this will only affect sn2, otherwise I'm interested in hearing feedback.

Yes, hp sx1000 systems represent interleaved memory in a separate
node w/o cpus.

Quoted 7 lines
Quoted 3 lines

So what's the ramifications of this patch? Does this mean that interleaved memory will now be preferably allocated over cell local memory on sx1000 systems?

Memory local to the node will come first, followed by the closest memory that doesn't have associated CPUs, followed by memory belonging to other CPUs.

Actually, I lied. There's a bug in find_next_best_node that prevents the local node from coming first if the PENALTY_FOR_NODE_WITH_CPUS is set high enough. Here's the fix (I'll send it to Andrew shortly).

Jesse

===== mm/page_alloc.c 1.239 vs edited =====

mm/page_alloc.c +6 -0
@@ -1213,6 +1213,12 @@
 		if (test_bit(n, used_node_mask))
 			continue;
 
+		/* Use the local node if we haven't already */
+		if (!test_bit(node, used_node_mask)) {
+			best_node = node;
+			break;
+		}
+
 		/* Use the distance array to find the distance */
 		val = node_distance(node, n);
 
#7 lore raw
In reply to Jesse Barnes
Quoted 8 lines · Jesse Barnes
Quoted 3 lines · Jesse Barnes

Memory local to the node will come first, followed by the closest memory that doesn't have associated CPUs, followed by memory belonging to other CPUs.

Actually, I lied. There's a bug in find_next_best_node that prevents the local node from coming first if the PENALTY_FOR_NODE_WITH_CPUS is set high enough. Here's the fix (I'll send it to Andrew shortly).

Great! Thanks for the review.

Alex