Skip to contentSkip to conversation

How can I optimize a process on a NUMA architecture(x86-64 specifically)?

DISCUSSION Brett E. 4 participants 10 messages May 21, 2004 – May 23, 2004

lore
Share & tools
lore

Brett E.

author
#0 lore raw

Say you have a bunch of single-threaded processes on a NUMA machine. Does the kernel make sure to prefer allocations using a certain CPU's memory, preferring to run a given process on the CPU which contains its memory? Or should I use the NUMA API(libnuma) to spell this out to the kernel? Does the kernel do the right thing in this case?

Thanks,

Brett

#1 lore raw
Quoted 4 lines · Brett E.

Say you have a bunch of single-threaded processes on a NUMA machine. Does the kernel make sure to prefer allocations using a certain CPU's memory, preferring to run a given process on the CPU which contains its memory?

Well, it'll allocate memory from the node containing the CPU that the process is running on, so if you've pinned your process (e.g. with schedutils) you'll be ok unless you're short on memory. If it's not pinned, you'll run the risk of having your process refer to memory on a remote node. Depending on what type of system you're running on, this could be a very small performance issue or a large one.

Quoted 2 lines

Or should I use the NUMA API(libnuma) to spell this out to the kernel? Does the kernel do the right thing in this case?

The kernel, by default, will allocate memory on the node where the process is running, and fall back to other nodes based on distance. That said, it's not a bad idea to pin your process to a CPU and use libnuma to explicitly set it's memory affinity.

Jesse

#2 lore raw
In reply to Brett E.
Quoted 5 lines

Say you have a bunch of single-threaded processes on a NUMA machine. Does the kernel make sure to prefer allocations using a certain CPU's memory, preferring to run a given process on the CPU which contains its memory? Or should I use the NUMA API(libnuma) to spell this out to the kernel? Does the kernel do the right thing in this case?

The kernel will generally do the right thing (process local alloc) by default. In 99% of cases, you don't want to muck with it - unless you're running one single app dominating the whole system, and nothing else is going on, you probably don't want to specify anything explicitly.

M.

author
#3 lore raw
Quoted 10 lines · Martin J. Bligh
Quoted 1 line

Say you have a bunch of single-threaded processes on a NUMA machine. Does the kernel make sure to prefer allocations using a certain CPU's memory, preferring to run a given process on the CPU which contains its memory? Or should I use the NUMA API(libnuma) to spell this out to the kernel? Does the kernel do the right thing in this case?

The kernel will generally do the right thing (process local alloc) by default. In 99% of cases, you don't want to muck with it - unless you're running one single app dominating the whole system, and nothing else is going on, you probably don't want to specify anything explicitly.

M.

Let's say I have a 2 way opteron and want to run 4 long-lived processes.
I fork and exec to create 1 of the processes, it chooses to run on
processor 0 since processor 1 is overloaded at that time, so its homenode is processor 0. I fork and exec another, it chooses processor 0 since processors 1 is overloaded at that time. .. Let's say an uneven distribution is chosen for all 4 processes, with all processes mapped to processor 0. So they allocate on node 0 yet the scheduler will map these to both processors since CPU should be balanced. In this case, you will have a situation where the second processor will have to fetch memory from the other processor's memory.

So a better solution would be to use numactl to set the homenodes explicitly, choosing processor 0 for 2 processes, processor 1 for the 2 other processes.

Is this incorrect?

#4 lore raw
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines

Say you have a bunch of single-threaded processes on a NUMA machine. Does the kernel make sure to prefer allocations using a certain CPU's memory, preferring to run a given process on the CPU which contains its memory? Or should I use the NUMA API(libnuma) to spell this out to the kernel? Does the kernel do the right thing in this case?

Quoted 8 lines
Quoted 7 lines

The kernel will generally do the right thing (process local alloc) by default. In 99% of cases, you don't want to muck with it - unless you're running one single app dominating the whole system, and nothing else is going on, you probably don't want to specify anything explicitly.

M.

Let's say I have a 2 way opteron and want to run 4 long-lived processes. I fork and exec to create 1 of the processes, it chooses to run on processor 0 since processor 1 is overloaded at that time, so its homenode is processor 0.

There is no such thing as a homenode. What you describe is more or less why we ditched that concept.

Quoted 1 line

I fork and exec another, it chooses processor 0 since processors 1 is overloaded at that time. .. Let's say an uneven distribution is chosen for all 4 processes, with all processes mapped to processor 0. So they allocate on node 0 yet the scheduler will map these to both processors since CPU should be balanced. In this case, you will have a situation where the second processor will have to fetch memory from the other processor's memory.

Each process will allocate local to the CPU it is running on when it does the allocate, so it's difficult to get as off-kilter as you describe (though it is still possible).

Quoted 1 line

So a better solution would be to use numactl to set the homenodes explicitly, choosing processor 0 for 2 processes, processor 1 for the 2 other processes.

In theory, it may be. If you ever had complete control of the system, and started no other processes whatsoever. In practice, that's very unlikely, so unless you're running a dedicated Oracle server or something, don't muck with it - just let the OS sort it out ;-)

M.

author
#5 lore raw
Quoted 18 lines · Martin J. Bligh
Quoted 8 lines
Quoted 8 lines
Quoted 1 line

Say you have a bunch of single-threaded processes on a NUMA machine. Does the kernel make sure to prefer allocations using a certain CPU's memory, preferring to run a given process on the CPU which contains its memory? Or should I use the NUMA API(libnuma) to spell this out to the kernel? Does the kernel do the right thing in this case?

The kernel will generally do the right thing (process local alloc) by default. In 99% of cases, you don't want to muck with it - unless you're running one single app dominating the whole system, and nothing else is going on, you probably don't want to specify anything explicitly.

Quoted 4 lines
Quoted 2 lines

M.

Let's say I have a 2 way opteron and want to run 4 long-lived processes. I fork and exec to create 1 of the processes, it chooses to run on processor 0 since processor 1 is overloaded at that time, so its homenode is processor 0.

There is no such thing as a homenode. What you describe is more or less why we ditched that concept.

Quoted 6 lines
Quoted 1 line

I fork and exec another, it chooses processor 0 since processors 1 is overloaded at that time. .. Let's say an uneven distribution is chosen for all 4 processes, with all processes mapped to processor 0. So they allocate on node 0 yet the scheduler will map these to both processors since CPU should be balanced. In this case, you will have a situation where the second processor will have to fetch memory from the other processor's memory.

Each process will allocate local to the CPU it is running on when it does the allocate, so it's difficult to get as off-kilter as you describe (though it is still possible).

So could process 0 run on processor 0, allocating local to processor 0, then run on processor 1, allocating local to processor 1, this way allocating to both processors? So over time process 0's allocations would be split up between both processors, defeating NUMA. The homenode concept + explicit CPU pinning seems useful in that they allow you to take advantage of NUMA better. Without these two things the kernel will just allocate on the currently running CPU whatever that may be when in fact a preference must be given to a CPU at some point, hopefully early on in the life of the process, in order to take advantage of NUMA.

I'm trying to play devil's advocate by the way so bear with me, you've been very helpful and I'm learning a great deal from you. :)

Thanks,

Brett

#6 lore raw
Quoted 1 line

So could process 0 run on processor 0, allocating local to processor 0, then run on processor 1, allocating local to processor 1, this way allocating to both processors? So over time process 0's allocations would be split up between both processors, defeating NUMA. The homenode concept + explicit CPU pinning seems useful in that they allow you to take advantage of NUMA better. Without these two things the kernel will just allocate on the currently running CPU whatever that may be when in fact a preference must be given to a CPU at some point, hopefully early on in the life of the process, in order to take advantage of NUMA.

For any given situation, you can come up with a scheduler mod that improves things. The problem is making something generic that works well in most cases.

I suggest: (1) read the archives. (2) Try implementing it if you still don't beleive me ;-) The main problem with the homenode scheduler is that you're more likely to end up in a situation where you're running "off-node" and allocating stuff back from your "home-node". This stuff isn't deterministic.

M.

#7 lore raw
In reply to Brett E.
Quoted 3 lines · Brett E.

So could process 0 run on processor 0, allocating local to processor 0, then run on processor 1, allocating local to processor 1, this way allocating to both processors?

Yep.

Quoted 2 lines

So over time process 0's allocations
would be split up between both processors, defeating NUMA.

Well, I wouldn't put it that way, but it would give your app suboptimal performance.

Quoted 3 lines

The homenode
concept + explicit CPU pinning seems useful in that they allow you to take advantage of NUMA better.

When you say homenode here, do you mean restricted memory allocation? If so, then yes, that would be useful. And we have it already with libnuma and sched_setaffinity. If you mean some sort of preferred allocation node, then no, we don't have that, and it wouldn't be of much benefit I think (relative to what we already have). The idea is to do "pretty good" by default, but still allow the user full control if they want to be sure.

Quoted 4 lines

Without these two things the kernel will
just allocate on the currently running CPU whatever that may be when in fact a preference must be given to a CPU at some point, hopefully early on in the life of the process, in order to take advantage of NUMA.

The kernel does a pretty good job of keeping processes on the CPU they've been running on, and thus they'll probably stay close to their memory. But without explicit pinning, there's no guarantee.

Jesse

#8 lore raw
Quoted 8 lines
Quoted 4 lines

Without these two things the kernel will
just allocate on the currently running CPU whatever that may be when in fact a preference must be given to a CPU at some point, hopefully early on in the life of the process, in order to take advantage of NUMA.

The kernel does a pretty good job of keeping processes on the CPU they've been running on, and thus they'll probably stay close to their memory. But without explicit pinning, there's no guarantee.

One thing I *do* want to do, is instead of explicit homenodes, is to take into account the per-node RSS of processes when doing my migration decisions. That gives us a kind of dynamic homenode. I have the per-node RSS code in my tree already from someone (Matt?), but we haven't used it. Of course, this throws off the O(1)-ness of balancing decisions, but as long as we cap the number we look at, and only do it for cross-node rebalance migrates (we don't need it for exec), I really don't care ;-)

M.

#9 lore raw
In reply to Brett E.
Quoted 12 lines

Let's say I have a 2 way opteron and want to run 4 long-lived processes.
I fork and exec to create 1 of the processes, it chooses to run on
processor 0 since processor 1 is overloaded at that time, so its homenode is processor 0. I fork and exec another, it chooses processor 0 since processors 1 is overloaded at that time. .. Let's say an uneven distribution is chosen for all 4 processes, with all processes mapped to processor 0. So they allocate on node 0 yet the scheduler will map these to both processors since CPU should be balanced. In this case, you will have a situation where the second processor will have to fetch memory from the other processor's memory.

Quoted 5 lines

So a better solution would be to use numactl to set the homenodes explicitly, choosing processor 0 for 2 processes, processor 1 for the 2 other processes.

Is this incorrect?

Generally, yes, it is. Surprisingly so. If you assume everything is
perfect, then this seems true. But in the real world, it almost never works that way.

Consider, for example, if process 1 is responsible for most of the memory
load at any particular time. If it's on CPU #1 and all its memory is on CPU #1, then the memory controller on CPU #2 is underused and memory bandwidth suffers. This is why most BIOSes interlave the memory pages across the CPUs. This gives you the best chance of being able to use both memory controllers under load.

I don't think we've reached the point yet where treating x86-64 systems as
NUMA machines makes very much sense.

DS