How can I optimize a process on a NUMA architecture(x86-64 specifically)?
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 3 lines · Martin J. Bligh
For any given situation, you can come up with a scheduler mod that improves things. The problem is making something generic that works well in most cases.
The point behind numa api/numactl is that if the defaults don't work well enough you can tune it by hand to be better.
There are some setups which can be significantly improved with some hand tuning, although in many cases the default behaviour is good enough too.
-Andi
Quoted 6 lines
Quoted 3 lines · Martin J. Bligh
For any given situation, you can come up with a scheduler mod that improves things. The problem is making something generic that works well in most cases.
Quoted 6 lines
The point behind numa api/numactl is that if the defaults don't work well enough you can tune it by hand to be better.
There are some setups which can be significantly improved with some hand tuning, although in many cases the default behaviour is good enough too.
Oh, I'm not denying it can make things better ... just 90% of the people who want to try it would be better off leaving it the hell alone ;-)
M.
Quoted 15 lines · Martin J. Bligh
Quoted 7 lines
Quoted 3 lines · Martin J. Bligh
For any given situation, you can come up with a scheduler mod that improves things. The problem is making something generic that works well in most cases.
Quoted 6 lines
The point behind numa api/numactl is that if the defaults don't work well enough you can tune it by hand to be better.
There are some setups which can be significantly improved with some hand tuning, although in many cases the default behaviour is good enough too.
Quoted 5 lines
Oh, I'm not denying it can make things better ... just 90% of the people who want to try it would be better off leaving it the hell alone ;-)
M.
Right now, 5 processes are running taking up a good deal of the CPU doing memory-intensive work(cacheing) and I notice that none of the processes seem to have CPU affinity. top shows they execute pseudo randomly on the CPU's.
At this point I'd like to decrease the number of processes to 4 and test performance with and without setting CPU & memory allocation affinity.
I've read the archives and I'm not sure how to get numactl running, both .5 and .6 versions give me:
# numactl --show No NUMA support available on this system.
despite using kernel 2.6.6.
running numactl under strace gives me:
32144 sched_getaffinity(32144, 16, { 0 }) = 8
32144 syscall_239(0, 0, 0, 0, 0, 0, 0x401e30, 0x401e30, 0x401e30, 0x401e30, 0x401e30, 0x401e30, 0x401e30, 0x401e30, 0x401e30, 0x401e30, 0x401e30, 0x401e30, 0x401e30, 0x401e30, 0x401e30, 0x401e30, 0x401e30, 0x401e30, 0x401e30, 0x401e30, 0x401e30, 0x401e30, 0x401e30, 0x401e30, 0x401e30, 0x401e30) = -1 (errno 38) 32144 fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 53), ...}) = 0
32144 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2a9556c000
32144 write(1, "No NUMA support available on thi"..., 42) = 42
32144 munmap(0x2a9556c000, 4096) = 0
32144 exit_group(1) = ?
In case someone might have ran into this before.
Thanks,
Brett
Quoted 23 lines
Right now, 5 processes are running taking up a good deal of the CPU doing memory-intensive work(cacheing) and I notice that none of the processes seem to have CPU affinity. top shows they execute pseudo randomly on the CPU's.
At this point I'd like to decrease the number of processes to 4 and test performance with and without setting CPU & memory allocation affinity.
I've read the archives and I'm not sure how to get numactl running, both .5 and .6 versions give me:
# numactl --show No NUMA support available on this system.despite using kernel 2.6.6.
running numactl under strace gives me:
32144
sched_getaffinity(32144, 16, { 0 })= 8
32144 syscall_239(0, 0, 0, 0, 0, 0,0x401e30,0x401e30,0x401e30,0x401e30,0x401e30,0x401e30,0x401e30,0x401e30,0x401e30,0x401e30,0x401e30,0x401e30,0x401e30,0x401e30,0x401e30,0x401e30,0x401e30,0x401e30,0x401e30,0x401e30,0x401e30,0x401e30,0x401e30,0x401e30,0x401e30,0x401e30) = -1 (errno 38) 32144fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 53), ...})= 0
32144mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0)=0x2a9556c000
32144write(1, "No NUMA support available on thi"..., 42)= 42
32144munmap(0x2a9556c000, 4096)= 0
32144exit_group(1)= ?
Quoted 1 line
In case someone might have ran into this before.
Did you turn on CONFIG_NUMA?
M.
Quoted 5 lines
Quoted 5 lines
# numactl --show No NUMA support available on this system.despite using kernel 2.6.6.
You need 2.6.6-mm*. numa api hasn't been merged to mainline yet.
-Andi
Quoted 3 lines · Brett E.
Right now, 5 processes are running taking up a good deal of the CPU doing memory-intensive work(cacheing) and I notice that none of the processes seem to have CPU affinity.
I don't know what kind of system you're running on, but if it's a multi-CPU Opteron, it is normally a sufficient fudge to just use sched_setaffinity to bind individual processes to specific CPUs. The mainline kernel memory allocator does the right thing in that case, and allocates memory locally when it can.
You can use the taskset command to get at this from the command line, so you may not even need to modify your code.
<b
Quoted 6 lines · Bryan O'Sullivan
Quoted 3 lines · Brett E.
Right now, 5 processes are running taking up a good deal of the CPU doing memory-intensive work(cacheing) and I notice that none of the processes seem to have CPU affinity.
Quoted 8 lines
I don't know what kind of system you're running on, but if it's a multi-CPU Opteron, it is normally a sufficient fudge to just use
sched_setaffinityto bind individual processes to specific CPUs. The mainline kernel memory allocator does the right thing in that case, and allocates memory locally when it can.You can use the taskset command to get at this from the command line, so you may not even need to modify your code.
Linus also merged the NUMA API support into mainline now with 2.6.7rc1, so you can use numactl for more finegrained tuning.
-Andi
Quoted 12 lines · Andi Kleen
Quoted 11 lines · Bryan O'Sullivan
Quoted 3 lines · Brett E.
Right now, 5 processes are running taking up a good deal of the CPU doing memory-intensive work(cacheing) and I notice that none of the processes seem to have CPU affinity.
I don't know what kind of system you're running on, but if it's a multi-CPU Opteron, it is normally a sufficient fudge to just use
sched_setaffinityto bind individual processes to specific CPUs. The mainline kernel memory allocator does the right thing in that case, and allocates memory locally when it can.
Quoted 5 lines
Quoted 2 lines
You can use the taskset command to get at this from the command line, so you may not even need to modify your code.
Linus also merged the NUMA API support into mainline now with 2.6.7rc1, so you can use numactl for more finegrained tuning.
FYI Brett, some Opteron systems have a BIOS option to interleave memory. If you are going to make use of NUMA, I think you want to not interleave.
Also, if you have a 25% imbalance within a domain/node, the scheduler can have a tendency to bounce around a task for fairness. That might be why you are seeing little/no affinity to a cpu (even top might be causing some of this). Not sure what the threshold is between domains/nodes, but I am curious if it still happens with CONFIG_NUMA on. If these are long lived cpu bound processes, I would try to have the number of processes be a multiple of the number of cpus.
-Andrew Theurer
Quoted 1 line · Andrew Theurer
FYI Brett, some Opteron systems have a BIOS option to interleave memory. If you are going to make use of NUMA, I think you want to not interleave.
I can confirm this. On my Tyan Thunder K8W 2885 (dual Opteron), I had to disable interleaving (under the Northbridge setup) before Linux recognized it as a NUMA system.
Signature · 3 lines
Scott Robert Ladd Coyote Gulch Productions (http://www.coyotegulch.com) Software Invention for High-Performance Computing
Quoted 16 lines · Andrew Theurer
Quoted 14 lines · Andi Kleen
Quoted 12 lines · Bryan O'Sullivan
Quoted 3 lines · Brett E.
Right now, 5 processes are running taking up a good deal of the CPU doing memory-intensive work(cacheing) and I notice that none of the processes seem to have CPU affinity.
I don't know what kind of system you're running on, but if it's a multi-CPU Opteron, it is normally a sufficient fudge to just use
sched_setaffinityto bind individual processes to specific CPUs. The mainline kernel memory allocator does the right thing in that case, and allocates memory locally when it can.
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
You can use the taskset command to get at this from the command line, so you may not even need to modify your code.
Linus also merged the NUMA API support into mainline now with 2.6.7rc1, so you can use numactl for more finegrained tuning.
FYI Brett, some Opteron systems have a BIOS option to interleave memory. If you are going to make use of NUMA, I think you want to not interleave.
Thanks for the heads up, I just disabled interleaving in the BIOS.
Quoted 2 lines
Also, if you have a 25% imbalance within a domain/node, the scheduler can have a tendency to bounce around a task for fairness. That might be why you are seeing little/no affinity to a cpu (even top might be causing some of this). Not sure what the threshold is between domains/nodes, but I am curious if it still happens with
CONFIG_NUMAon. If these are long lived cpu bound processes, I would try to have the number of processes be a multiple of the number of cpus.
I have CONFIG_NUMA on and yes these are long-lived processes(duration is 1 hour). And you and I are on the same wavelength, I'm imagining 3 processes per CPU with apache handing off work to the processes in question. This is on a 1.6ghz 2-way opteron if that matters at all. Hopefully I can make the modifications and test this tomorrow.
Thanks,
Brett
Quoted 2 lines · Martin J. Bligh
There is no such thing as a homenode. What you describe is more or less why we ditched that concept.
numa api has a "prefered node", which is a bit similar to the old home node. The main difference is that it does not affect the scheduler, only the memory allocation. You can of course affect the scheduler too, but that's a separate option now and more strict.
For historical reasons numactl still has a --homenode= alias for --prefered, although it is undocumented and discouraged now.
-Andi
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 2 lines · David Schwartz
I don't think we've reached the point yet where treating x86-64 systems as
NUMA machines makes very much sense.
Benchmarks disagree with you on that. In most cases local memory policy seems to work better than BIOS interleaving. That's because memory latency is usually more important than memory bandwidth.
-Andi