Skip to contentSkip to conversation

How can I optimize a process on a NUMA architecture(x86-64 specifically)?

DISCUSSION Martin J. Bligh 6 participants 12 messages May 21, 2004 – May 25, 2004 7 messages not in the archive

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

#6 lore raw
Quoted 3 lines · Martin J. Bligh

For any given situation, you can come up with a scheduler mod that improves things. The problem is making something generic that works well in most cases.

The point behind numa api/numactl is that if the defaults don't work well enough you can tune it by hand to be better.

There are some setups which can be significantly improved with some hand tuning, although in many cases the default behaviour is good enough too.

-Andi

#7 lore raw
Quoted 6 lines
Quoted 3 lines · Martin J. Bligh

For any given situation, you can come up with a scheduler mod that improves things. The problem is making something generic that works well in most cases.

Quoted 6 lines

The point behind numa api/numactl is that if the defaults don't work well enough you can tune it by hand to be better.

There are some setups which can be significantly improved with some hand tuning, although in many cases the default behaviour is good enough too.

Oh, I'm not denying it can make things better ... just 90% of the people who want to try it would be better off leaving it the hell alone ;-)

M.

#8 lore raw
Quoted 15 lines · Martin J. Bligh
Quoted 7 lines
Quoted 3 lines · Martin J. Bligh

For any given situation, you can come up with a scheduler mod that improves things. The problem is making something generic that works well in most cases.

Quoted 6 lines

The point behind numa api/numactl is that if the defaults don't work well enough you can tune it by hand to be better.

There are some setups which can be significantly improved with some hand tuning, although in many cases the default behaviour is good enough too.

Quoted 5 lines

Oh, I'm not denying it can make things better ... just 90% of the people who want to try it would be better off leaving it the hell alone ;-)

M.

Right now, 5 processes are running taking up a good deal of the CPU doing memory-intensive work(cacheing) and I notice that none of the processes seem to have CPU affinity. top shows they execute pseudo randomly on the CPU's.

At this point I'd like to decrease the number of processes to 4 and test performance with and without setting CPU & memory allocation affinity.

I've read the archives and I'm not sure how to get numactl running, both .5 and .6 versions give me:

# numactl --show
No NUMA support available on this system.

despite using kernel 2.6.6.

running numactl under strace gives me:

32144 sched_getaffinity(32144, 16, { 0 }) = 8
32144 syscall_239(0, 0, 0, 0, 0, 0, 0x401e30, 0x401e30, 0x401e30, 0x401e30, 0x401e30, 0x401e30, 0x401e30, 0x401e30, 0x401e30, 0x401e30, 0x401e30, 0x401e30, 0x401e30, 0x401e30, 0x401e30, 0x401e30, 0x401e30, 0x401e30, 0x401e30, 0x401e30, 0x401e30, 0x401e30, 0x401e30, 0x401e30, 0x401e30, 0x401e30) = -1 (errno 38) 32144 fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 53), ...}) = 0
32144 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2a9556c000
32144 write(1, "No NUMA support available on thi"..., 42) = 42
32144 munmap(0x2a9556c000, 4096) = 0
32144 exit_group(1) = ?

In case someone might have ran into this before.

Thanks,

Brett

#9 lore raw
Quoted 23 lines

Right now, 5 processes are running taking up a good deal of the CPU doing memory-intensive work(cacheing) and I notice that none of the processes seem to have CPU affinity. top shows they execute pseudo randomly on the CPU's.

At this point I'd like to decrease the number of processes to 4 and test performance with and without setting CPU & memory allocation affinity.

I've read the archives and I'm not sure how to get numactl running, both .5 and .6 versions give me:

# numactl --show
No NUMA support available on this system.

despite using kernel 2.6.6.

running numactl under strace gives me:

32144 sched_getaffinity(32144, 16, { 0 }) = 8
32144 syscall_239(0, 0, 0, 0, 0, 0, 0x401e30, 0x401e30, 0x401e30, 0x401e30, 0x401e30, 0x401e30, 0x401e30, 0x401e30, 0x401e30, 0x401e30, 0x401e30, 0x401e30, 0x401e30, 0x401e30, 0x401e30, 0x401e30, 0x401e30, 0x401e30, 0x401e30, 0x401e30, 0x401e30, 0x401e30, 0x401e30, 0x401e30, 0x401e30, 0x401e30) = -1 (errno 38) 32144 fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 53), ...}) = 0
32144 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2a9556c000
32144 write(1, "No NUMA support available on thi"..., 42) = 42
32144 munmap(0x2a9556c000, 4096) = 0
32144 exit_group(1) = ?

Quoted 1 line

In case someone might have ran into this before.

Did you turn on CONFIG_NUMA?

M.

#10 lore raw
Quoted 5 lines
Quoted 5 lines
# numactl --show
No NUMA support available on this system.

despite using kernel 2.6.6.

You need 2.6.6-mm*. numa api hasn't been merged to mainline yet.

-Andi

#11 lore raw
In reply to Brett E.
Quoted 3 lines · Brett E.

Right now, 5 processes are running taking up a good deal of the CPU doing memory-intensive work(cacheing) and I notice that none of the processes seem to have CPU affinity.

I don't know what kind of system you're running on, but if it's a multi-CPU Opteron, it is normally a sufficient fudge to just use sched_setaffinity to bind individual processes to specific CPUs. The mainline kernel memory allocator does the right thing in that case, and allocates memory locally when it can.

You can use the taskset command to get at this from the command line, so you may not even need to modify your code.

	<b
#12 lore raw
Quoted 6 lines · Bryan O'Sullivan
Quoted 3 lines · Brett E.

Right now, 5 processes are running taking up a good deal of the CPU doing memory-intensive work(cacheing) and I notice that none of the processes seem to have CPU affinity.

Quoted 8 lines

I don't know what kind of system you're running on, but if it's a multi-CPU Opteron, it is normally a sufficient fudge to just use sched_setaffinity to bind individual processes to specific CPUs. The mainline kernel memory allocator does the right thing in that case, and allocates memory locally when it can.

You can use the taskset command to get at this from the command line, so you may not even need to modify your code.

Linus also merged the NUMA API support into mainline now with 2.6.7rc1, so you can use numactl for more finegrained tuning.

-Andi

#13 lore raw
Quoted 12 lines · Andi Kleen
Quoted 11 lines · Bryan O'Sullivan
Quoted 3 lines · Brett E.

Right now, 5 processes are running taking up a good deal of the CPU doing memory-intensive work(cacheing) and I notice that none of the processes seem to have CPU affinity.

I don't know what kind of system you're running on, but if it's a multi-CPU Opteron, it is normally a sufficient fudge to just use sched_setaffinity to bind individual processes to specific CPUs. The mainline kernel memory allocator does the right thing in that case, and allocates memory locally when it can.

Quoted 5 lines
Quoted 2 lines

You can use the taskset command to get at this from the command line, so you may not even need to modify your code.

Linus also merged the NUMA API support into mainline now with 2.6.7rc1, so you can use numactl for more finegrained tuning.

FYI Brett, some Opteron systems have a BIOS option to interleave memory. If you are going to make use of NUMA, I think you want to not interleave.

Also, if you have a 25% imbalance within a domain/node, the scheduler can have a tendency to bounce around a task for fairness. That might be why you are seeing little/no affinity to a cpu (even top might be causing some of this). Not sure what the threshold is between domains/nodes, but I am curious if it still happens with CONFIG_NUMA on. If these are long lived cpu bound processes, I would try to have the number of processes be a multiple of the number of cpus.

-Andrew Theurer

#14 lore raw
Quoted 1 line · Andrew Theurer

FYI Brett, some Opteron systems have a BIOS option to interleave memory. If you are going to make use of NUMA, I think you want to not interleave.

I can confirm this. On my Tyan Thunder K8W 2885 (dual Opteron), I had to disable interleaving (under the Northbridge setup) before Linux recognized it as a NUMA system.

Signature · 3 lines
Scott Robert Ladd
Coyote Gulch Productions (http://www.coyotegulch.com)
Software Invention for High-Performance Computing
#15 lore raw
In reply to Andrew Theurer
Quoted 16 lines · Andrew Theurer
Quoted 14 lines · Andi Kleen
Quoted 12 lines · Bryan O'Sullivan
Quoted 3 lines · Brett E.

Right now, 5 processes are running taking up a good deal of the CPU doing memory-intensive work(cacheing) and I notice that none of the processes seem to have CPU affinity.

I don't know what kind of system you're running on, but if it's a multi-CPU Opteron, it is normally a sufficient fudge to just use sched_setaffinity to bind individual processes to specific CPUs. The mainline kernel memory allocator does the right thing in that case, and allocates memory locally when it can.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

You can use the taskset command to get at this from the command line, so you may not even need to modify your code.

Linus also merged the NUMA API support into mainline now with 2.6.7rc1, so you can use numactl for more finegrained tuning.

FYI Brett, some Opteron systems have a BIOS option to interleave memory. If you are going to make use of NUMA, I think you want to not interleave.

Thanks for the heads up, I just disabled interleaving in the BIOS.

Quoted 2 lines

Also, if you have a 25% imbalance within a domain/node, the scheduler can have a tendency to bounce around a task for fairness. That might be why you are seeing little/no affinity to a cpu (even top might be causing some of this). Not sure what the threshold is between domains/nodes, but I am curious if it still happens with CONFIG_NUMA on. If these are long lived cpu bound processes, I would try to have the number of processes be a multiple of the number of cpus.

I have CONFIG_NUMA on and yes these are long-lived processes(duration is 1 hour). And you and I are on the same wavelength, I'm imagining 3 processes per CPU with apache handing off work to the processes in question. This is on a 1.6ghz 2-way opteron if that matters at all. Hopefully I can make the modifications and test this tomorrow.

Thanks,

Brett

#16 lore raw
In reply to a message not in the archive
Quoted 2 lines · Martin J. Bligh

There is no such thing as a homenode. What you describe is more or less why we ditched that concept.

numa api has a "prefered node", which is a bit similar to the old home node. The main difference is that it does not affect the scheduler, only the memory allocation. You can of course affect the scheduler too, but that's a separate option now and more strict.

For historical reasons numactl still has a --homenode= alias for --prefered, although it is undocumented and discouraged now.

-Andi

A message that is not in the archive (sent off-list or not yet received): [email protected]

#18 lore raw
Quoted 2 lines · David Schwartz

I don't think we've reached the point yet where treating x86-64 systems as
NUMA machines makes very much sense.

Benchmarks disagree with you on that. In most cases local memory policy seems to work better than BIOS interleaving. That's because memory latency is usually more important than memory bandwidth.

-Andi