NUMA allocator on Opteron systems does non-local allocation on node0
Quoted 4 lines
I've notived that the memory allocation on NUMA systems (Opterons) does memory allocation on non-local nodes for processes running node0 even if local memory is available. (Kernel 2.6.25 and above)
How much local memory is available? 8GB per node? That means there will be 4GB on node 0 in ZONE_DMA32 and 4GB in ZONE_NORMAL. Other nodes will have 8GB in ZONE_NORMAL.
Quoted 8 lines
In my setup I'm allocating an array of ~7GiB memory size in a singlethreaded application.
Startup: numactl --cpunodebind=X ./app
For X=1,2,3 it works as expected, all memory is allocated on the local node.
For X=0 I can see the memory beeing allocated on node0 as long as ~3GiB are "free" on node0. At this point the kernel starts using memory from node1 for the app!
NUMA only supports memory policies for the highest zone which is ZONE_NORMAL here. Only 4GB of ZONE_NORMAL are available on node 0, so it will go off node after that memory is exhausted. This is done in order to preserve the lower 4GB for I/O to 32 bit devices.
Hi,
this problem/question is allready solved for me. Andi suggested to post this on the linux-mm mailing list and they helped me. :)
Quoted 11 lines
Quoted 5 lines
I've notived that the memory allocation on NUMA systems (Opterons) does
memory allocation on non-local nodes for processes running node0 even if
local memory is available. (Kernel 2.6.25 and above)How much local memory is available? 8GB per node? That means there will be 4GB
on node 0 inZONE_DMA32and 4GB inZONE_NORMAL. Other nodes will have 8GB inZONE_NORMAL.
You're right. This machine has 8GiB per node. Due to the memory hole the machine has ~3GiB ZONE_DMA32 which perfectly fits to my observations.
Quoted 12 lines
Quoted 11 lines
In my setup I'm allocating an array of ~7GiB memory size in a singlethreaded application.
Startup: numactl --cpunodebind=X ./app
For X=1,2,3 it works as expected, all memory is allocated on the local
node.
For X=0 I can see the memory beeing allocated on node0 as long as ~3GiB
are "free" on node0. At this point the kernel starts using memory from
node1 for the app!
Quoted 6 lines
NUMA only supports memory policies for the highest zone which is
ZONE_NORMALhere. Only 4GB ofZONE_NORMALare available on node 0, so it will
go off node after that memory is exhausted. This is done in order to preserve
the lower 4GB for I/O to 32 bit devices.
I've changed the policy from "default" to "node" (/proc/sys/vm/numa_zonelist_order) and now it works fine for me. Policy "default" does automaticly select "node" or "zone" depending on the machine. When the policy is set to "default" the kernel (2.6.27) chooses "node" if
1. there is no ZONE_DMA32
2. the size of ZONE_DMA32 is greater than 50% of the system memory
3. the size of ZONE_DMA32 is greater than 60% of the nodelocal memory
Signature · 2 lines
Regards, Oliver Weihe