prefer allocations from nodes w/o CPUs
Subsystems (2)
RFC 1/1 prefer allocations from nodes w/o CPUs
Small patch for the ia64 version of topology.h to make the zonelist building code prefer nodes w/o CPUs for allocations. Do any platforms aside from Altix have nodes with only memory? If not, then this will only affect sn2, otherwise I'm interested in hearing feedback.
Thanks,
Jesse
===== include/asm-ia64/topology.h 1.13 vs edited =====
include/asm-ia64/topology.h +4 -0
@@ -18,6 +18,10 @@#include <asm/smp.h> #ifdef CONFIG_NUMA + +/* Nodes w/o CPUs are preferred for memory allocations, see build_zonelists */ +#define PENALTY_FOR_NODE_WITH_CPUS 255 +/* * Returns the number of the node containing CPU 'cpu' */
Quoted 4 lines · Jesse Barnes
Small patch for the ia64 version of
topology.hto make the zonelist building code prefer nodes w/o CPUs for allocations. Do any platforms aside from Altix have nodes with only memory? If not, then this will only affect sn2, otherwise I'm interested in hearing feedback.
Yes, hp sx1000 systems represent interleaved memory in a separate
node w/o cpus.
Alex
Signature · 1 line
Alex Williamson HP Linux & Open Source Lab
Quoted 8 lines · Alex Williamson
Quoted 4 lines · Jesse Barnes
Small patch for the ia64 version of
topology.hto make the zonelist building code prefer nodes w/o CPUs for allocations. Do any platforms aside from Altix have nodes with only memory? If not, then this will only affect sn2, otherwise I'm interested in hearing feedback.Yes, hp sx1000 systems represent interleaved memory in a separate
node w/o cpus.
So does this patch make sense for those platforms too? Or do we need something fancier? It'll make nodes w/o CPUs come before nodes with CPUs in the zonelist (after the local node of course), even if the nodes with CPUs are closer.
Jesse
Quoted 7 lines · Jesse Barnes
Quoted 6 lines · Alex Williamson
Quoted 4 lines · Jesse Barnes
Small patch for the ia64 version of
topology.hto make the zonelist building code prefer nodes w/o CPUs for allocations. Do any platforms aside from Altix have nodes with only memory? If not, then this will only affect sn2, otherwise I'm interested in hearing feedback.
Quoted 7 lines
Quoted 2 lines
Yes, hp sx1000 systems represent interleaved memory in a separate
node w/o cpus.So does this patch make sense for those platforms too? Or do we need something fancier? It'll make nodes w/o CPUs come before nodes with CPUs in the zonelist (after the local node of course), even if the nodes with CPUs are closer.
That makes sense for us. The interleaved memory should be somewhat
faster than truly remote memory. Depending on the box configuration, the interleaved memory node could be quite small and we'll run out quickly, but better to make what use we can from it. Thanks,
Alex
Signature · 1 line
Alex Williamson HP Linux & Open Source Lab
Quoted 8 lines · Alex Williamson
Quoted 4 lines · Jesse Barnes
Small patch for the ia64 version of
topology.hto make the zonelist building code prefer nodes w/o CPUs for allocations. Do any platforms aside from Altix have nodes with only memory? If not, then this will only affect sn2, otherwise I'm interested in hearing feedback.Yes, hp sx1000 systems represent interleaved memory in a separate
node w/o cpus.
So what's the ramifications of this patch? Does this mean that interleaved memory will now be preferably allocated over cell local memory on sx1000 systems?
Thanks,
Alex Tsariounov
Quoted 7 lines · Alex Tsariounov
Quoted 6 lines · Alex Williamson
Quoted 4 lines · Jesse Barnes
Small patch for the ia64 version of
topology.hto make the zonelist building code prefer nodes w/o CPUs for allocations. Do any platforms aside from Altix have nodes with only memory? If not, then this will only affect sn2, otherwise I'm interested in hearing feedback.
Quoted 6 lines
Quoted 2 lines
Yes, hp sx1000 systems represent interleaved memory in a separate
node w/o cpus.So what's the ramifications of this patch? Does this mean that interleaved memory will now be preferably allocated over cell local memory on sx1000 systems?
Memory local to the node will come first, followed by the closest memory that doesn't have associated CPUs, followed by memory belonging to other CPUs.
Jesse
Quoted 12 lines · Jesse Barnes
Quoted 11 lines · Alex Tsariounov
Quoted 9 lines · Alex Williamson
Quoted 5 lines · Jesse Barnes
Small patch for the ia64 version of
topology.hto make the zonelist building code prefer nodes w/o CPUs for allocations. Do any platforms aside from Altix have nodes with only memory? If not, then this will only affect sn2, otherwise I'm interested in hearing feedback.Yes, hp sx1000 systems represent interleaved memory in a separate
node w/o cpus.
Quoted 7 lines
Quoted 3 lines
So what's the ramifications of this patch? Does this mean that interleaved memory will now be preferably allocated over cell local memory on sx1000 systems?
Memory local to the node will come first, followed by the closest memory that doesn't have associated CPUs, followed by memory belonging to other CPUs.
Actually, I lied. There's a bug in find_next_best_node that prevents the local node from coming first if the PENALTY_FOR_NODE_WITH_CPUS is set high enough. Here's the fix (I'll send it to Andrew shortly).
Jesse
===== mm/page_alloc.c 1.239 vs edited =====
mm/page_alloc.c +6 -0
@@ -1213,6 +1213,12 @@if (test_bit (n, used_node_mask))continue ; +/* Use the local node if we haven't already */ +if (!test_bit (node, used_node_mask)) { + best_node = node; +break ; + } +/* Use the distance array to find the distance */ val =node_distance (node, n);
Quoted 8 lines · Jesse Barnes
Quoted 3 lines · Jesse Barnes
Memory local to the node will come first, followed by the closest memory that doesn't have associated CPUs, followed by memory belonging to other CPUs.
Actually, I lied. There's a bug in
find_next_best_nodethat prevents the local node from coming first if thePENALTY_FOR_NODE_WITH_CPUSis set high enough. Here's the fix (I'll send it to Andrew shortly).
Great! Thanks for the review.
Alex