v2: KVM-userspace: add NUMA support for guests

Hi,

this patch series introduces multiple NUMA nodes support within KVM guests. This is the second try incorporating several requests from the list:
- use the QEMU firmware configuration interface instead of CMOS-RAM
- detect presence of libnuma automatically, can be disabled with ./configure --disable-numa
This only applies to the host side, the command line and guest (BIOS) side are always built and functional, although this configuration is only useful for research and debugging
- use a more flexible command line interface allowing:
- specifying the distribution of memory across the guest nodes:

    mem:1536M;512M
  - specifying the distribution of the CPUs:
    cpu:0-2;3
  - specifying the host nodes the guest nodes should be pinned to:
    pin:3;2

All of these options are optional, in case of mem and cpu the resources are split equally across all guest nodes if omitted. Please note that at least in Linux SRAT takes precedence over E820, so the total usable memory will be the sum specified at the mem: option (although QEMU will still allocate the amount at -m). If pin: is omitted, the guest nodes will be pinned to those host nodes where the threads are happen to be scheduled at on start-up time. This requires the (v)getcpu (v)syscall to be usable, this is true for kernels up from 2.6.19 and glibc >= 2.6 (sched_getcpu()). I have a hack if glibc doesn't support this, tell me if you are interested. The only non-optional argument is the number of guest nodes, a possible command line looks like:
-numa 3,mem:1024M;512M;512M,cpu:0-1;2;3
Please note that you have to quote the semicolons on the shell.

The monitor command is left out for now and will be send later.

Please apply.

Regards,
Andre.

Signed-off-by Andre Przywara <[email protected]>

This patch series needs to be posted to qemu-devel. I know qemu doesn't do true SMP yet, but it will in the relatively near future. Either way, some of the design points needs review from a larger audience than present on kvm-devel.

I'm not a big fan of the libnuma dependency. I'll willing to concede this if there's a wide agreement that we should support this directly in QEMU.

I don't think there's such a thing as a casual NUMA user. The default NUMA policy in Linux is node-local memory. As long as a VM is smaller than a single node, everything will work out fine.

In the event that the VM is larger than a single node, if a user is creating it via qemu-system-x86_64, they're going to either not care at all about NUMA, or be familiar enough with the numactl tools that they'll probably just want to use that. Once you've got your head around the fact that VCPUs are just threads and the memory is just a shared memory segment, any knowledgable sysadmin will have no problem doing whatever sort of NUMA layout they want.

The other case is where management tools are creating VMs. In this case, it's probably better to use numactl as an external tool because then it keeps things consistent wrt CPU pinning.

There's also a good argument for not introducing CPU pinning directly to QEMU. There are multiple ways to effectively do CPU pinning. You can use taskset, you can use cpusets or even something like libcgroup.

If you refactor the series so that the libnuma patch is the very last one and submit to qemu-devel, I'll review and apply all of the first patches. We can continue to discuss the last patch independently of the first three if needed.

Quoted 32 lines · Andre Przywara

Hi,

this patch series introduces multiple NUMA nodes support within KVM guests. This is the second try incorporating several requests from the list:
- use the QEMU firmware configuration interface instead of CMOS-RAM
- detect presence of libnuma automatically, can be disabled with ./configure --disable-numa
This only applies to the host side, the command line and guest (BIOS) side are always built and functional, although this configuration is only useful for research and debugging
- use a more flexible command line interface allowing:
- specifying the distribution of memory across the guest nodes:

    mem:1536M;512M
  - specifying the distribution of the CPUs:
    cpu:0-2;3
  - specifying the host nodes the guest nodes should be pinned to:
    pin:3;2

All of these options are optional, in case of mem and cpu the resources are split equally across all guest nodes if omitted. Please note that at least in Linux SRAT takes precedence over E820, so the total usable memory will be the sum specified at the mem: option (although QEMU will still allocate the amount at -m). If pin: is omitted, the guest nodes will be pinned to those host nodes where the threads are happen to be scheduled at on start-up time. This requires the (v)getcpu (v)syscall to be usable, this is true for kernels up from 2.6.19 and glibc >= 2.6 (sched_getcpu()). I have a hack if glibc doesn't support this, tell me if you are interested. The only non-optional argument is the number of guest nodes, a possible command line looks like:
-numa 3,mem:1024M;512M;512M,cpu:0-1;2;3
Please note that you have to quote the semicolons on the shell.

The monitor command is left out for now and will be send later.

Please apply.

Regards,
Andre.

Signed-off-by Andre Przywara <[email protected]>

In the event that the VM is larger than a single node, if a user is creating it via qemu-system-x86_64, they're going to either not care at all about NUMA, or be familiar enough with the numactl tools that they'll probably just want to use that. Once you've got your head around the fact that VCPUs are just threads and the memory is just a shared memory segment, any knowledgable sysadmin will have no problem doing whatever sort of NUMA layout they want.

The other case is where management tools are creating VMs. In this case, it's probably better to use numactl as an external tool because then it keeps things consistent wrt CPU pinning.

There's also a good argument for not introducing CPU pinning directly to QEMU. There are multiple ways to effectively do CPU pinning. You can use taskset, you can use cpusets or even something like libcgroup.

If you refactor the series so that the libnuma patch is the very last one and submit to qemu-devel, I'll review and apply all of the first patches. We can continue to discuss the last patch independently of the first three if needed.

Quoted 3 lines · Anthony Liguori

In the event that the VM is larger than a single node, if a user is creating it via qemu-system-x86_64, they're going to either not care at all about NUMA, or be familiar enough with the numactl tools that they'll probably just want to use that. Once you've got your head around the fact that VCPUs are just threads and the memory is just a shared memory segment, any knowledgable sysadmin will have no problem doing whatever sort of NUMA layout they want.

The vast majority of production VMs will be created by management tools.

We need libnuma integrated in qemu. Using numactl outside of qemu means we need to start exposing more and more qemu internals (vcpu->thread mapping, memory in /dev/shm, phys_addr->ram_addr mapping) and lose out on optimization opportunities (having multiple numa-aware iothreads, numa-aware kvm mmu). It also means we cause duplication of the numa logic in management tools instead of consolidation in qemu.

Anthony,

Quoted 1 line

This patch series needs to be posted to qemu-devel. I know qemu doesn't do true SMP yet, but it will in the relatively near future. Either way, some of the design points needs review from a larger audience than present on kvm-devel.

OK, I already started looking at that. The first patch applies with only some fuzz, so no problems here. The second patch could be changed to promote the values via the firmware configuration interface only, leaving the host side pinning alone (which wouldn't make much sense without true SMP anyway). The third patch is actually BOCHS BIOS, and I am confused here:
I see the host side of the firmware config interface in QEMU SVN, but neither in the BOCHS CVS nor in the qemu/pc-bios/bios.diff there is any sign of usage from the BIOS side.

Is the kvm-patched qemu the only user of the interface? If so I would have to introduce the interface to QEMU's bios.diff (or better send to bochs-developers?) Do you know what BOCHS version the bios.diff applies against? Is that the 2.3.7 release?

Quoted 1 line

I'm not a big fan of the libnuma dependency. I'll willing to concede this if there's a wide agreement that we should support this directly in QEMU.

As long as QEMU is not true SMP, libnuma is rather useless. One could pin the memory to the appropriate host nodes, but without the proper scheduling this doesn't make much sense. And rescheduling the qemu process each time a new VCPU is scheduled doesn't seem so smart, either.

Quoted 1 line

I don't think there's such a thing as a casual NUMA user. The default NUMA policy in Linux is node-local memory. As long as a VM is smaller than a single node, everything will work out fine.

Almost right, but simply calling qemu-system-x86_64 can lead to bad situations. I lately saw that VCPU #0 was scheduled on one node and VCPU #1 on another. This leads to random (probably excessive) remote accesses from the VCPUs, since the guest assumes uniform memory

Of course one could cure this small guest case with numactl, but in my experience the existence of this tool isn't as well-known as one would expect.

Quoted 2 lines

In the event that the VM is larger than a single node, if a user is creating it via qemu-system-x86_64, they're going to either not care at all about NUMA, or be familiar enough with the numactl tools that they'll probably just want to use that. Once you've got your head around the fact that VCPUs are just threads and the memory is just a shared memory segment, any knowledgable sysadmin will have no problem doing whatever sort of NUMA layout they want.

Really? How do you want to assign certain _parts_ of guest memory with numactl? (Let alone the rather weird way of using -mempath, which is much easier done within QEMU).

The same applies to the threads. You can assign _all_ the threads to certain nodes, but pinning single threads only requires some tedious work (QEMU monitor or top, then taskset -p). Isn't that OK if qemu would do this automatically (or at least give some support here)?

Quoted 3 lines

The other case is where management tools are creating VMs. In this case, it's probably better to use numactl as an external tool because then it keeps things consistent wrt CPU pinning.

There's also a good argument for not introducing CPU pinning directly to QEMU. There are multiple ways to effectively do CPU pinning. You can use taskset, you can use cpusets or even something like libcgroup.

I agree that pinning isn't the last word on the subject, but it works pretty well. But I wouldn't load the admin with the burden of pinning, but let this be done by QEMU/KVM. Maybe one could introduce a way to tell QEMU/KVM to not pin the threads.

I also had the idea to start with some sort of pinning (either automatically or user-chosen) and lift the affinity later (after the thread has done something and touched some memory). In this case Linux could (but probably will not easily) move the thread to another node. One could think about triggering this from a management app: If the app detects a congestion on one node, it could first lift the affinity restriction of some VCPU threads to achieve a better load balancing. If the situation persists (and doesn't turn out to be a short time peak), the manager could migrate the memory too and pin the VCPUs to the new node. I thought the migration and temporary un-pinning could be implemented in the monitor.

Quoted 3 lines

...
The third patch is actually BOCHS BIOS, and I am confused here:
I see the host side of the firmware config interface in QEMU SVN, but neither in the BOCHS CVS nor in the qemu/pc-bios/bios.diff there is any sign of usage from the BIOS side.

Really? I assumed it was there. I'll look this afternoon and if it isn't, I'll apply those patches to bios.diff and update the bios.

Quoted 1 line

Quoted 1 line

I'm not a big fan of the libnuma dependency. I'll willing to concede this if there's a wide agreement that we should support this directly in QEMU.

Quoted 1 line

As long as QEMU is not true SMP, libnuma is rather useless....

Even if it's not useful, I'd still like to add it to QEMU. That's one less thing that has to be merged from KVM into QEMU.

Quoted 1 line

Almost right, but simply calling qemu-system-x86_64 can lead to bad situations. I lately saw that VCPU #0 was scheduled on one node and VCPU #1 on another. This leads to random (probably excessive) remote accesses from the VCPUs, since the guest assumes uniform memory

That seems like Linux is behaving badly, no? Can you describe the situation more?

NUMA systems are expensive. If a customer cares about performance (as opposed to just getting more memory), then I think tools like numactl are pretty well known.

Quoted 2 lines

...
Really? How do you want to assign certain _parts_ of guest memory with numactl? (Let alone the rather weird way of using -mempath, which is much easier done within QEMU).

I don't think -mem-path is weird at all. In fact, I'd be inclined to use shared memory by default and create a temporary file name. Then provide a monitor interface to lookup that file name so that an explicit -mem-path isn't required anymore.

Quoted 2 lines

...
But I wouldn't load the admin with the burden of pinning, but let this be done by QEMU/KVM. Maybe one could introduce a way to tell QEMU/KVM to not pin the threads.

This is where things start to get ugly...

The other issue with pinning is what happens after live migration? What about single-machine load balancing? Regardless of whether we bake in libnuma control or not, I think an interface on the command line is not terribly interesting because it's too static.

I think a monitor
interface is what we'd really want if we integrated with libnuma.

info numa

numa pin:0;3

numa pin:*;

numa migrate:1;2

I was partly wrong, the code is in BOCHS CVS, but not in qemu. It wasn't in BOCHS 2.3.7 release, which qemu is currently based on. Could you pull the latest BIOS code from BOCHS CVS to qemu? This would give us the firmware interface for free and I could more easily port my patches.

What's actually bothering you with libnuma dependency? I could directly use the Linux mbind syscall, but I think using a library is more sane (and probably more portable).

Quoted 2 lines

Quoted 1 line

Almost right, but simply calling qemu-system-x86_64 can lead to bad situations. I lately saw that VCPU #0 was scheduled on one node and VCPU #1 on another. This leads to random (probably excessive) remote accesses from the VCPUs, since the guest assumes uniform memory

That seems like Linux is behaving badly, no? Can you describe the situation more?

That is just my observation. I have to do more research to get a decent explanation, but I think the problem is that in this early state the threads barely touch any memory, so Linux tries to distribute them as best as possible. Just a quick run on a quad node machine with 16 cores in total:

Quoted 1 line

NUMA systems are expensive. If a customer cares about performance (as opposed to just getting more memory), then I think tools like numactl are pretty well known.

Well, expensive depends, especially if I think of your employer ;-) In fact every AMD dual socket server is NUMA, and Intel will join the game next year.

Additionally one could use some kind of home node, so one temporarily could change the VCPUs affinity and later return to the optimal affinity (where the memory is located) without specifying it again.

Comments are welcome.

Regards,
Andre.

Quoted 2 lines

Quoted 1 line

But I wouldn't load the admin with the burden of pinning, but let this be done by QEMU/KVM. Maybe one could introduce a way to tell QEMU/KVM to not pin the threads.

This is where things start to get ugly...

Why? qemu-system-x86_64 -numa 2,pin:none and then use whatever method you prefer (taskset, monitor) to pin the VCPUs (or left them unpinned).

// similar to numactl --hardware, * means all nodes (no pinning)

Quoted 1 line

numa pin:0;3

// static pinning: guest 0 -> host 0, guest 1 -> host 3

Quoted 1 line

numa pin:*;

// guest node 0 -> all nodes, guest node 1: keep as it is // or maybe: numa pin:0-3;

Quoted 1 line

numa migrate:1;2

Quoted 3 lines · André Przywara

Quoted 2 lines

Quoted 1 line

But I wouldn't load the admin with the burden of pinning, but let this be done by QEMU/KVM. Maybe one could introduce a way to tell QEMU/KVM to not pin the threads.

This is where things start to get ugly...

Why? qemu-system-x86_64 -numa 2,pin:none and then use whatever method you prefer (taskset, monitor) to pin the VCPUs (or left them unpinned).

I agree that for e.g. -numa 2, no host binding should occur. Pinning memory or cpus to nodes should only occur if the user explicitly requested it. Otherwise we run the risk of breaking load balancing.

If the user chooses to pin, the responsibility is on them. If not, we should allow the host to do its thing.