NUMA support in HVM guests

-----Original Message-----
From: [email protected] [mailto:xen-devel- [email protected]] On Behalf Of Andre Przywara
Sent: Monday, August 13, 2007 6:01 PM
To: [email protected]
Subject: [Xen-devel] [PATCH 0/4] [HVM] NUMA support in HVM guests

Hi,

these four patches allow to forward NUMA characteristics into HVM guests. This works by allocating memory explicitly from different NUMA nodes and create an appropriate SRAT-ACPI table which describes the topology. Needs a decent guest kernel which uses the SRAT table to discover the NUMA topology.
This allows to break the current de-facto limitation of guests to one NUMA node, one can use more memory and/or more VCPUs than there are available on one node.

Patch 1/4: introduce numanodes=n config file option.
this states how many NUMA nodes the guest should see, the default is
0,
which means to turn off most parts of the code.
Patch 2/4: introduce CPU affinity for allocate_physmap call.
currently
the correct NUMA node to take the memory from is chosen by simply using the currently scheduled CPU, this patch allows to explicitly specify a CPU and provides XENMEM_DEFAULT_CPU for the old behavior
Patch 3/4: allocate memory with NUMA in mind.
actually look at the numanodes=n option to split the memory request up into n parts and allocate it from different nodes. Also change the VCPUs affinity to match the nodes.
Patch 4/4: inject created SRAT table into the guest.
create a SRAT table, fill it up with the desired NUMA topology and inject it into the guest

Applies against staging c/s #15719.

Signed-off-by Andre Przywara <[email protected]>

Regards,
Andre.

--
Andre Przywara
AMD-Operating System Research Center (OSRC), Dresden, Germany
Tel: +49 351 277-84917
----to satisfy European Law for business letters:
AMD Saxony Limited Liability Company & Co. KG
Sitz (Geschäftsanschrift): Wilschdorfer Landstr. 101, 01109 Dresden, Deutschland
Registergericht Dresden: HRA 4896
vertretungsberechtigter Komplementär: AMD Saxony LLC (Sitz Wilmington, Delaware, USA)
Geschäftsführer der AMD Saxony LLC: Dr. Hans-R. Deppe, Thomas McCoy

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

(quoted diff) +5 -0

+    for (i=0;i<=dominfo.max_vcpu_id;i++)
+    {
+        node= ( i * numanodes ) / (dominfo.max_vcpu_id+1);
+        xc_vcpu_setaffinity (xc_handle, dom, i, nodemasks[node]);
+    }

This always starts from node0, this may make node0 very busy, while other nodes may not have many work.

It may be nice to pin node from the lightest overhead node.

We also need to add some limitations for numanodes. The number of vcpus on vnode should not be larger than the number of pcpus on pnode. Otherwise vcpus belonging to a domain run
on the same pcpu, which is not what we want.

In setup_numa_mem, each node has even memory size, if the memory allocation fails, >the domain creation fails. This may be too "rude", I think we can

NUMA with each node has different memory size, even more, and maybe some node doesn't have memory. What we need guarantee is guest see physical topology.

In your patch, when create NUMA guest, vnode is pinned to pnode. While after some creations and destroys domain operation, the workload on the platform may be very imbalanced, we need a method to dynamically balance workload. There are two methods IMO.
1. Implement NUMA-aware scheduler and page migration
2. Run a daemon in dom0, this daemon monitors workload, and use live-migration to balance workload if necessary.

Quoted 2 lines

This always starts from node0, this may make node0 very busy, while

nodes may not have many work.
This is true, I encountered this before, but didn't want to wait longer for sending up the patches. Actually the "numanodes=n" config file option shouldn't specify the number of nodes, but a list of specific nodes to use, like "numanodes=0,2" to pin the domain on the first and the third node.

Quoted 1 line

We also need to add some limitations for numanodes. The number of

on vnode should not be larger

Quoted 1 line

than the number of pcpus on pnode. Otherwise vcpus belonging to a

domain run

Quoted 1 line

on the same pcpu, which is not what we want.

Would be nice, but in the moment I would push this into the sysadmin's responsibility.

After all my patches were more a discussion base than a final solution, so I see there is more work to do. In the moment I am working on including PV guests.

Quoted 2 lines

This always starts from node0, this may make node0 very busy, while

nodes may not have many work.
This is true, I encountered this before, but didn't want to wait longer for sending up the patches. Actually the "numanodes=n" config file option shouldn't specify the number of nodes, but a list of specific nodes to use, like "numanodes=0,2" to pin the domain on the first and the third node.

Quoted 1 line

We also need to add some limitations for numanodes. The number of

on vnode should not be larger

Quoted 1 line

than the number of pcpus on pnode. Otherwise vcpus belonging to a

domain run

Quoted 1 line

on the same pcpu, which is not what we want.

Would be nice, but in the moment I would push this into the sysadmin's responsibility.

After all my patches were more a discussion base than a final solution, so I see there is more work to do. In the moment I am working on including PV guests.

I read your patches and Anthony's commands. Write a patch based on

1: If guest set numanodes=n (default it will be 1 means that this
guest will be restricted in one node); hypervisor will choose
begin node to pin for this guest use round robin. But the method I use
need a spin_lock to prevent create domain at same time. Are there any
more good methods, hope for your suggestion.

2: pass node parameter use higher bits in flags when create domain.
At this time, domain can record node information in domain struct
for further use, i.e. show which node to pin when setup_guest.
If use this method, in your patch, can simply balance nodes just
like below;

Review

Quoted 6 lines

(quoted diff) +6 -0

+    for (i=0;i<=dominfo.max_vcpu_id;i++)
+    {
+        node= ( i * numanodes ) / (dominfo.max_vcpu_id+1)+ 		
+		domaininfo.first_node;
+        xc_vcpu_setaffinity (xc_handle, dom, i, nodemasks[node]);
+    }

thanks Ronghui for your patches and ideas. To make a more structured approach to a better NUMA support, I suggest to concentrate on one-node-guests first:

* introduce CPU affinity to memory allocation routines called from

This is basically my patch 2/4 from August. We should think about using a NUMA node number instead of a physical CPU, is there something to be said against this?

* find _some_ method of load balancing when creating guests. The method 1 from Ronghui is a start, but a real decision based on each node's utilization (or free memory) would be more reasonable.

* patch the guest memory allocation routines to allocate memory from that specific node only (based on my patch 3/4)

* use live migration to local host to allow node migration. Assuming that localhost live migration works reliably (is that really true?) it shouldn't be too hard to implement this (basically just using node affinity while allocating guest memory). Since this is a rather expensive operation (takes twice the memory temporarily and quite some time), I'd suggest to trigger that explicitly from the admin via a xm command, maybe as an addition to migrate:

# xm migrate --live --node 1 <domid> localhost
There could be some Dom0 daemon based re-balancer to do this somewhat
automatically later on.

I would take care of the memory allocation patch and would look into node migration. It would be great if Roughui or Anthony would help to improve the "load balancing" algorithm.

Meanwhile I will continue to patch that d*** Linux kernel to accept

CONFIG_NUMA and CONFIG_XEN without crashing that early ;-), this should allow both HVM and PV guests to support multiple NUMA nodes within one guest.

Also we should start a discussion on the config file options to add:
Shall we use "numanodes=<nr of nodes>", something like "numa=on" (for one-node-guests only), or something like "numanode=0,1" to explicitly specify certain nodes?

Any comments are appreciated.

Quoted 5 lines

I read your patches and Anthony's commands. Write a patch based on

1: If guest set numanodes=n (default it will be 1 means that this
guest will be restricted in one node); hypervisor will choose
begin node to pin for this guest use round robin. But the

Quoted 1 line

need a spin_lock to prevent create domain at same time. Are

Quoted 1 line

more good methods, hope for your suggestion.

That's a good start, thank you. Maybe Keir has some comments on the spinlock issue.

Quoted 6 lines

2: pass node parameter use higher bits in flags when create domain.
At this time, domain can record node information in domain struct
for further use, i.e. show which node to pin when setup_guest.
If use this method, in your patch, can simply balance nodes just
like below;

Quoted 6 lines

Review

Quoted 6 lines

(quoted diff) +6 -0

+    for (i=0;i<=dominfo.max_vcpu_id;i++)
+    {
+        node= ( i * numanodes ) / (dominfo.max_vcpu_id+1)+
+		domaininfo.first_node;
+        xc_vcpu_setaffinity (xc_handle, dom, i, nodemasks[node]);
+    }

How many bits do you want to use? Maybe it's not a good idea to abuse some variable to hold a limited number of nodes only ("640K ought to be enough for anybody" ;-) But the general idea is good.