Xen 3.4.1 NUMA support
Hello,
does the last version of Xen(3.4.1) support NUMA machines? Is there a .pdf or a link that can give me some more details about that? I work on a project for xen performace in numa machines. And in xen 3.3.0 this performance isn't good. Have something changed in last version?
Thanks in advance,
Papagiannis Anastasios
Add Xen boot parameter 'numa=on' to enable NUMA detection. Then it's up to you to, for example, pin domains to specific nodes, using the 'cpus=...' option in the domain config file. See /etc/xen/xmexample1 for an example of its usage.
-- Keir
Quoted 15 lines · Papagiannis Anastasios
Hello,
does the last version of Xen(3.4.1) support NUMA machines? Is there a .pdf or a link that can give me some more details about that? I work on a project for xen performace in numa machines. And in xen 3.3.0 this performance isn't good. Have something changed in last version?
Thanks in advance,
Papagiannis Anastasios_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel
VMware has the notion of a "cell" where VMs can be scheduled only within a cell, not across cells. Cell boundaries are determined by VMware by default, though certains settings can override them.
An interesting project might be to implement "numa=cell" for Xen.... or maybe something similar is already in George Dunlap's scheduler plans?
Quoted 44 lines
-----Original Message-----
From: Keir Fraser [mailto:[email protected]]
Sent: Wednesday, November 04, 2009 5:33 AM
To: Papagiannis Anastasios; [email protected]
Subject: Re: [Xen-devel] Xen 3.4.1 NUMA supportAdd Xen boot parameter 'numa=on' to enable NUMA detection. Then it's up to
you to, for example, pin domains to specific nodes, using the 'cpus=...'
option in the domain config file. See/etc/xen/xmexample1for an example of
its usage.-- Keir
Quoted 3 lines · Papagiannis Anastasios
Hello,
does the last version of Xen(3.4.1) support NUMA machines?
Is there a .pdf
Quoted 4 lines
or a link that can give me some more details about that? I work on a project for xen performace in numa machines. And in xen 3.3.0 this performance isn't good. Have something changed in last version?
Quoted 8 lines
Thanks in advance,
Papagiannis Anastasios_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel
I haven't had time to look at NUMA stuff at all. I probably will look at it eventually, if no one else does, but I'd be happy if someone else could pursue it.
-George
Quoted 53 lines · Dan Magenheimer
VMware has the notion of a "cell" where VMs can be scheduled only within a cell, not across cells. Cell boundaries are determined by VMware by default, though certains settings can override them.
An interesting project might be to implement "numa=cell" for Xen.... or maybe something similar is already in George Dunlap's scheduler plans?
Quoted 43 lines
-----Original Message-----
From: Keir Fraser [mailto:[email protected]]
Sent: Wednesday, November 04, 2009 5:33 AM
To: Papagiannis Anastasios; [email protected]
Subject: Re: [Xen-devel] Xen 3.4.1 NUMA supportAdd Xen boot parameter 'numa=on' to enable NUMA detection. Then it's up to you to, for example, pin domains to specific nodes, using the 'cpus=...' option in the domain config file. See
/etc/xen/xmexample1for an example of its usage.-- Keir
Quoted 4 lines · Papagiannis Anastasios
Hello,
does the last version of Xen(3.4.1) support NUMA machines?
Is there a .pdf
Quoted 4 lines
or a link that can give me some more details about that? I work on a project for xen performace in numa machines. And in xen 3.3.0 this performance isn't good. Have something changed in last version?
Quoted 9 lines
Thanks in advance,
Papagiannis Anastasios_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel
George,
What's the current scope and status of your scheduler work ? Is it going to look similar to the Linux scheduler (with scheduling domains, et al). In that case, topology is already accounted for, to a large extent. It would be good to know so that I can work on something that doesn't overlap.
-dulloor
Quoted 74 lines · George Dunlap
I haven't had time to look at NUMA stuff at all. I probably will look at it eventually, if no one else does, but I'd be happy if someone else could pursue it.
-George
Quoted 60 lines · Dan Magenheimer
VMware has the notion of a "cell" where VMs can be scheduled only within a cell, not across cells. Cell boundaries are determined by VMware by default, though certains settings can override them.
An interesting project might be to implement "numa=cell" for Xen.... or maybe something similar is already in George Dunlap's scheduler plans?
Quoted 49 lines
-----Original Message-----
From: Keir Fraser [mailto:[email protected]]
Sent: Wednesday, November 04, 2009 5:33 AM
To: Papagiannis Anastasios; [email protected]
Subject: Re: [Xen-devel] Xen 3.4.1 NUMA supportAdd Xen boot parameter 'numa=on' to enable NUMA detection. Then it's up to
you to, for example, pin domains to specific nodes, using the 'cpus=...' option in the domain config file. See/etc/xen/xmexample1for an example of
its usage.-- Keir
Quoted 4 lines · Papagiannis Anastasios
Hello,
does the last version of Xen(3.4.1) support NUMA machines?
Is there a .pdf
Quoted 5 lines
or a link that can give me some more details about that? I work on a project for xen performace in numa machines. And in xen 3.3.0 this performance isn't good. Have something changed in last version?
Quoted 9 lines
Thanks in advance,
Papagiannis Anastasios_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel
Quoted 5 lines · Dulloor
What's the current scope and status of your scheduler work ? Is it going to look similar to the Linux scheduler (with scheduling domains, et al). In that case, topology is already accounted for, to a large extent. It would be good to know so that I can work on something that doesn't overlap.
My plan was to do something similar to Linux, but with this
difference: Instead of having one runqueue per logical processor (as both Xen and Linux currently do), and having "domains" all the way up (as Linux currently does), I had planned on having one runqueue per L2 processor cache. The main reason to avoid migration is to preserve a warm cache; but since L1's are replaced so quickly, there should be little impact to a VM migrating between different threads and cores which share the same L2.
Above the L2s I was planning on having an idea similar to the Linux "domains" (although obviously it would need a different name to avoid confusion), and doing explicit load-balancing between them. But as I have not had a chance to test this kind of load balancing yet, the plan may change somewhate before then.
Problems to solve wrt NUMA, as I understand it, are to balance the performance cost of sharing a busy local CPU, vs the performance cost of non-local memory accesses. This would involve adding the NUMA logic to the load balancing algorithm. Which I guess would depend in part on having a load balancing algorithm to begin with. :-)
Once I have the basic credit patches in working order, would you be interested in working on the load-balancing between runqueues? I can then work on further testing of the credit algorithm. My ultimate goal would be to have a basic regression test that people could use to measure how their changes to the scheduler affect a wide variety of workloads.
-George
Sure ! Let know when you have the patches ready. Also, that might be a good time to see if runq-per-l2 works better.
-dulloor
Quoted 37 lines · George Dunlap
Quoted 5 lines · Dulloor
What's the current scope and status of your scheduler work ? Is it going to look similar to the Linux scheduler (with scheduling domains, et al). In that case, topology is already accounted for, to a large extent. It would be good to know so that I can work on something that doesn't overlap.
My plan was to do something similar to Linux, but with this
difference: Instead of having one runqueue per logical processor (as both Xen and Linux currently do), and having "domains" all the way up (as Linux currently does), I had planned on having one runqueue per L2 processor cache. The main reason to avoid migration is to preserve a warm cache; but since L1's are replaced so quickly, there should be little impact to a VM migrating between different threads and cores which share the same L2.Above the L2s I was planning on having an idea similar to the Linux "domains" (although obviously it would need a different name to avoid confusion), and doing explicit load-balancing between them. But as I have not had a chance to test this kind of load balancing yet, the plan may change somewhate before then.
Problems to solve wrt NUMA, as I understand it, are to balance the performance cost of sharing a busy local CPU, vs the performance cost of non-local memory accesses. This would involve adding the NUMA logic to the load balancing algorithm. Which I guess would depend in part on having a load balancing algorithm to begin with. :-)
Once I have the basic credit patches in working order, would you be interested in working on the load-balancing between runqueues? I can then work on further testing of the credit algorithm. My ultimate goal would be to have a basic regression test that people could use to measure how their changes to the scheduler affect a wide variety of workloads.
-George
Cpupools? :-)
NUMA was a topic I wanted to look at as soon as cpupools are officially accepted. Keir wanted to propose a way to get rid of the function continue_hypercall_on_cpu() which was causing most of the stuff leading to the objection of cpupools.
I guess Keir had some higher priority jobs. :-)
So I will try a new patch for cpupools without continue_hypercall_on_cpu() and perhaps with NUMA support.
George, would this be okay for you? I think your scheduler still will have problems with domain weights as long as domains are restricted to some processors, right?
Juergen
Quoted 70 lines · George Dunlap
I haven't had time to look at NUMA stuff at all. I probably will look at it eventually, if no one else does, but I'd be happy if someone else could pursue it.
-George
Quoted 5 lines · Dan Magenheimer
VMware has the notion of a "cell" where VMs can be scheduled only within a cell, not across cells. Cell boundaries are determined by VMware by default, though certains settings can override them.
Quoted 50 lines
An interesting project might be to implement "numa=cell" for Xen.... or maybe something similar is already in George Dunlap's scheduler plans?
Quoted 45 lines
-----Original Message-----
From: Keir Fraser [mailto:[email protected]]
Sent: Wednesday, November 04, 2009 5:33 AM
To: Papagiannis Anastasios; [email protected]
Subject: Re: [Xen-devel] Xen 3.4.1 NUMA supportAdd Xen boot parameter 'numa=on' to enable NUMA detection. Then it's up to
you to, for example, pin domains to specific nodes, using the 'cpus=...' option in the domain config file. See/etc/xen/xmexample1for an example of
its usage.-- Keir
Quoted 3 lines · Papagiannis Anastasios
Hello,
does the last version of Xen(3.4.1) support NUMA machines?
Is there a .pdf
Quoted 4 lines
or a link that can give me some more details about that? I work on a project for xen performace in numa machines. And in xen 3.3.0 this performance isn't good. Have something changed in last version?
Quoted 9 lines
Thanks in advance,
Papagiannis Anastasios_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel
Signature · 5 lines
Juergen Gross Principal Developer Operating Systems TSP ES&S SWE OS6 Telephone: +49 (0) 89 636 47950 Fujitsu Technolgy Solutions e-mail: [email protected] Otto-Hahn-Ring 6 Internet: ts.fujitsu.com D-81739 Muenchen Company details: ts.fujitsu.com/imprint.html
Quoted 3 lines · Juergen Gross
George, would this be okay for you? I think your scheduler still will have problems with domain weights as long as domains are restricted to some processors, right?
Hmm, this may be a point of discussion at some point.
My plan was actually to have one runqueue per L2 processor cache. Thus as many as 4 cores (and possibly 8 hyperthreads) would be sharing the same runqueue; doing CPU pinning within the same runqueue would be problematic.
I was planning on having credits work mainly within one runqueue, and then do load balancing between runqueues. In that case pinning to a specific runqueue shouldn't cause a problem, because credits of one runqueue wouldn't affect credtis of another one.
However, I haven't implemented or tested this idea yet; it's possible that having credits kept distinct and doing load balancing between runqueues will cause unacceptable levels of unfairness. I expect it to be fine (esp since Linux's scheduler does this kind of load balancing, but doesn't share runqueues between logical processors), but without implementation and testing I can't say for sure.
Thoughts are welcome at this point, but it will probably be better to have a real discussion once I've posted some patches.
-George
Quoted 5 lines · Juergen Gross
NUMA was a topic I wanted to look at as soon as cpupools are officially accepted. Keir wanted to propose a way to get rid of the function
continue_hypercall_on_cpu()which was causing most of the stuff leading to the objection of cpupools.
I guess Keir had some higher priority jobs. :-)
Well, I forgot about it. I think the plan was to perhaps keep something like continue_hypercall_on_cpu(), but not need to actually run the vcpu itself 'over there' but instead schedule a tasklet or somesuch, and sleep on its completion. That would get rid of the skanky affinity hacks you had to do to support continue_hypercall_on_cpu(). I'll have a look back at what we discussed.
-- Keir
Quoted 6 lines · Dan Magenheimer
Quoted 2 lines
Add Xen boot parameter 'numa=on' to enable NUMA detection. Then it's up to you to, for example, pin domains to specific nodes, using the 'cpus=...' option in the domain config file. See
/etc/xen/xmexample1for an example of its usage.VMware has the notion of a "cell" where VMs can be scheduled only within a cell, not across cells. Cell boundaries are determined by VMware by default, though certains settings can override them.
Well, If I got this right, then you are describing the current behaviour of Xen. It has a similar feature for some time now (since 3.3, I guess). When you launch a domain on a numa=on machine, it will pick the least busiest node (which can hold the requested memory) and restrict the domain to that node (by only allowing CPUs of that node). This is in XendDomainInfo.py (c/s 17131, 17247, 17709)
Looks like this one:
(kernel xen.gz numa=on dom0_mem=6144M dom0_max_vcpus=6 dom0_vcpus_pin)
# xm create opensuse.hvm # xm create opensuse2.hvm # xm vcpu-list Name ID VCPU CPU State Time(s) CPU Affinity 001-LTP 1 0 6 -b- 17.8 6-11 001-LTP 1 1 7 -b- 6.3 6-11 002-LTP 2 0 12 -b- 19.0 12-17 002-LTP 2 1 16 -b- 1.6 12-17 002-LTP 2 2 17 -b- 1.7 12-17 002-LTP 2 3 14 -b- 1.6 12-17 002-LTP 2 4 16 -b- 1.6 12-17 002-LTP 2 5 15 -b- 1.5 12-17 002-LTP 2 6 12 -b- 1.3 12-17 002-LTP 2 7 13 -b- 1.8 12-17 Domain-0 0 0 0 -b- 12.6 0 Domain-0 0 1 1 -b- 7.6 1 Domain-0 0 2 2 -b- 8.0 2 Domain-0 0 3 3 -b- 14.6 3 Domain-0 0 4 4 r-- 1.4 4 Domain-0 0 5 5 -b- 0.9 5 # xm debug-keys U (XEN) Domain 0 (total: 2097152): (XEN) Node 0: 2097152 (XEN) Node 1: 0 (XEN) Node 2: 0 (XEN) Node 3: 0 (XEN) Node 4: 0 (XEN) Node 5: 0 (XEN) Node 6: 0 (XEN) Node 7: 0 (XEN) Domain 1 (total: 394219): (XEN) Node 0: 0 (XEN) Node 1: 394219 (XEN) Node 2: 0 (XEN) Node 3: 0 (XEN) Node 4: 0 (XEN) Node 5: 0 (XEN) Node 6: 0 (XEN) Node 7: 0 (XEN) Domain 2 (total: 394219): (XEN) Node 0: 0 (XEN) Node 1: 0 (XEN) Node 2: 394219 (XEN) Node 3: 0 (XEN) Node 4: 0 (XEN) Node 5: 0 (XEN) Node 6: 0 (XEN) Node 7: 0
Note that there were no cpus= lines in the config files, Xen did that automatically.
Domains can be localhost-migrated to another node:
# xm migrate --node=4 1 localhost The only issue is with domains larger than a node. If someone has a useful use-case, I can start rebasing my old patches for NUMA aware HVM domains to Xen unstable.
Regards,
Andre.
BTW: Shouldn't we set finally numa=on as the default value?
Signature · 3 lines
Andre Przywara AMD-OSRC (Dresden) Tel: x29712
Quoted 2 lines · Andre Przywara
BTW: Shouldn't we set finally numa=on as the default value?
Is there any data to support the idea that this helps significantly on common systems?
-George
Quoted 4 lines · George Dunlap
Quoted 2 lines · Andre Przywara
BTW: Shouldn't we set finally numa=on as the default value?
Is there any data to support the idea that this helps significantly on common systems?
I don't have any numbers handy, but I will try if I can generate some.
Looking from a high level perspective it is a shame that it's not the default: With numa=off the Xen domain loader will allocate physical memory from some node (maybe even from several nodes) and will schedule the guest on some other (even rapidly changing) nodes. According to Murphy's law you will end up with _all_ the memory access of a guest to be remote. But in fact a NUMA architecture is really beneficial for virtualization: As there are close to zero cross domain memory accesses (except for Dom0), each node is more or less self contained and each guest can use the node's memory controller almost exclusively. But this is all spoiled as most people don't know about Xen's NUMA capabilities and don't set numa=on. Using this as a default would solve this.
Regards,
Andre.
Signature · 9 lines
Andre Przywara AMD-Operating System Research Center (OSRC), Dresden, Germany Tel: +49 351 488-3567-12 ----to satisfy European Law for business letters: Advanced Micro Devices GmbH Karl-Hammerschmidt-Str. 34, 85609 Dornach b. Muenchen Geschaeftsfuehrer: Jochen Polster; Thomas M. McCoy; Giuliano Meroni Sitz: Dornach, Gemeinde Aschheim, Landkreis Muenchen Registergericht Muenchen, HRB Nr. 43632
I am not finding this. Can you please point to the code ?
numa=on/off is only for setting up numa in xen (similar to the linux knob, but turned off by default). The allocation of memory from a single node (that you observe) could be because of the way alloc_heap_pages is implemented (trying to allocate from all the heaps from a node, before trying the next one) - try looking at dump_numa output. And, affinities are not set anywhere based on the node from which allocation happens.
-dulloor
Quoted 45 lines · Andre Przywara
Quoted 8 lines · George Dunlap
Quoted 3 lines · Andre Przywara
BTW: Shouldn't we set finally numa=on as the default value?
Is there any data to support the idea that this helps significantly on common systems?
I don't have any numbers handy, but I will try if I can generate some.
Looking from a high level perspective it is a shame that it's not the
default: With numa=off the Xen domain loader will allocate physical memory from some node (maybe even from several nodes) and will schedule the guest on some other (even rapidly changing) nodes. According to Murphy's law you will end up with _all_ the memory access of a guest to be remote. But in fact a NUMA architecture is really beneficial for virtualization: As there are close to zero cross domain memory accesses (except for Dom0), each node is more or less self contained and each guest can use the node's memory controller almost exclusively.
But this is all spoiled as most people don't know about Xen's NUMA capabilities and don't set numa=on. Using this as a default would solve this.Regards,
Andre.--
Andre Przywara
AMD-Operating System Research Center (OSRC), Dresden, Germany
Tel: +49 351 488-3567-12
----to satisfy European Law for business letters:
Advanced Micro Devices GmbH
Karl-Hammerschmidt-Str. 34, 85609 Dornach b. Muenchen
Geschaeftsfuehrer: Jochen Polster; Thomas M. McCoy; Giuliano Meroni
Sitz: Dornach, Gemeinde Aschheim, Landkreis Muenchen Registergericht Muenchen, HRB Nr. 43632_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel
Quoted 1 line · Dulloor
I am not finding this. Can you please point to the code ?
tools/python/xen/xend/XendDomainInfo.py (around line 2600)
with the core code being:
-------------
index = nodeload.index( min(nodeload) )
cpumask = info['node_to_cpu'][index]
for v in range(0, self.info['VCPUs_max']):
xc.vcpu_setaffinity(self.domid, v, cpumask)
--------------
The code got introduced with c/s 17131 and later got refined with c/s 17247 and c/s 17709.
Quoted 6 lines
numa=on/off is only for setting up numa in xen (similar to the linux knob, but turned off by default). The allocation of memory from a single node (that you observe) could be because of the way
alloc_heap_pagesis implemented (trying to allocate from all the heaps from a node, before trying the next one)
Yes, but if the domain is pinned before it allocated it's memory, then the natural behavior of Xen is to take memory from this local node.
Quoted 3 lines
- try looking at
dump_numa
output. And, affinities are not set anywhere based on the node from which allocation happens.
It is the other way round, first the domain is pinned, later the memory is allocated (based on the node to which the currently scheduled CPU is belonging to).
Regards,
Andre.
Quoted 28 lines
-dulloor
Quoted 24 lines · Andre Przywara
Quoted 5 lines · George Dunlap
Quoted 2 lines · Andre Przywara
BTW: Shouldn't we set finally numa=on as the default value?
Is there any data to support the idea that this helps significantly on common systems?
I don't have any numbers handy, but I will try if I can generate some.
Looking from a high level perspective it is a shame that it's not the
default: With numa=off the Xen domain loader will allocate physical memory from some node (maybe even from several nodes) and will schedule the guest on some other (even rapidly changing) nodes. According to Murphy's law you will end up with _all_ the memory access of a guest to be remote. But in fact a NUMA architecture is really beneficial for virtualization: As there are close to zero cross domain memory accesses (except for Dom0), each node is more or less self contained and each guest can use the node's memory controller almost exclusively.
But this is all spoiled as most people don't know about Xen's NUMA capabilities and don't set numa=on. Using this as a default would solve this.Regards,
Andre.
Signature · 9 lines
Andre Przywara AMD-Operating System Research Center (OSRC), Dresden, Germany Tel: +49 351 448 3567 12 ----to satisfy European Law for business letters: Advanced Micro Devices GmbH Karl-Hammerschmidt-Str. 34, 85609 Dornach b. Muenchen Geschaeftsfuehrer: Andrew Bowd; Thomas M. McCoy; Giuliano Meroni Sitz: Dornach, Gemeinde Aschheim, Landkreis Muenchen Registergericht Muenchen, HRB Nr. 43632
Quoted 4 lines · George Dunlap
Quoted 2 lines · Andre Przywara
BTW: Shouldn't we set finally numa=on as the default value?
Is there any data to support the idea that this helps significantly on common systems?
I did some tests on an 8 node machine. I will retry this later on 4-nodes and 2-nodes systems, but I assume similar numbers. I used multiple guests in parallel each running bw_mem of lmbench, which is admittedly quite NUMA sensitive. I cannot publish real numbers (yet?), but the results were dramatic:
with numa=on I got the same results for each guest (the same as the native result) when the number of guests was smaller or equal the number of nodes (since each guest got it's own memory controller).
If I disabled NUMA aware placement by explicitly specifying cpus="0-31"
in the config file or booted with numa=off, the values dropped down by
factor 3-5 (!) (even for a few guests) with some variance due to the random nature of core to memory mapping.
Overcommitting the nodes (letting multiple guests use each node) lowered the values to about 80% for two guests and 60% for three guests per node, but it never got anywhere close to the numa=off values.
So these results encourage me again to opt for numa=on as the default value.
Keir, I will check if dropping the node containment in the CPU overcommitment case is an option, but what would be the right strategy in that case?
Warn the user?
Don't contain at all?
Contain to more than onde node?
Regards,
Andre.
Signature · 9 lines
Andre Przywara AMD-Operating System Research Center (OSRC), Dresden, Germany Tel: +49 351 448 3567 12 ----to satisfy European Law for business letters: Advanced Micro Devices GmbH Karl-Hammerschmidt-Str. 34, 85609 Dornach b. Muenchen Geschaeftsfuehrer: Andrew Bowd; Thomas M. McCoy; Giuliano Meroni Sitz: Dornach, Gemeinde Aschheim, Landkreis Muenchen Registergericht Muenchen, HRB Nr. 43632
Quoted 11 lines
Overcommitting the nodes (letting multiple guests use each node) lowered the values to about 80% for two guests and 60% for three guests per node, but it never got anywhere close to the numa=off values. So these results encourage me again to opt for numa=on as the default value.
Keir, I will check if dropping the node containment in the CPU overcommitment case is an option, but what would be the right strategy in that case?
Warn the user?
Don't contain at all?
Contain to more than onde node?
In the case where a VM is asking for more vCPUs there are pCPUs in a node we should contain the guest to multiple nodes. (I presume we favour nodes according to the number of vCPUs they already have committed to them?)
We should turn off automatic node containment of any kind if the total number of pCPUs in the system is <= 8 -- on such systems the statistical multiplexing gain of having access to more pCPUs likely outweighs the NUMA placement benefit and memory striping will be a better strategy. I'm inclined to believe that may be true for 2 node systems with <=16 pCPUs too under many workloads
I'd really like to see us enumerate pCPUs in a sensible order so that it's easier to see the topology. It should be nodes.sockets.cores{.threads}, leaving gaps for missing execution units due to hot plug or non power of two packing. Right now we're inconsistent in the enumeration order depending on how the BIOS has set things up. It would be great if someone could volunteer to fix this...
Ian
Quoted 1 line · Ian Pratt
In the case where a VM is asking for more vCPUs there are pCPUs in a node we should contain the guest to multiple nodes. (I presume we favour nodes according to the number of vCPUs they already have committed to them?)
Seems like CPU load might be a better measure. Xen doesn't calculate load currently, but it's on my list of things to do.
-George
Quoted 8 lines
Quoted 1 line · Ian Pratt
In the case where a VM is asking for more vCPUs there are pCPUs in a
node we should contain the guest to multiple nodes. (I presume we favour nodes according to the number of vCPUs they already have committed to them?)
Seems like CPU load might be a better measure. Xen doesn't calculate load currently, but it's on my list of things to do.
I'd rather get this stuff fixed now than wait for the new scheduler.
It's not clear that instantaneous CPU load is any better than just counting the number of vCPUs. The XCP xapi stack also records good historical data, and would be in a better position to do the placement. Further work.
Ian
Quoted 7 lines · Ian Pratt
I'd really like to see us enumerate pCPUs in a sensible order so that it's easier to see the topology. It should be nodes.sockets.cores{.threads}, leaving gaps for missing execution units due to hot plug or non power of two packing.
Right now we're inconsistent in the enumeration order depending on how the BIOS has set things up. It would be great if someone could volunteer to fix this...
Even better would be to have pCPUs addressable and listable explicitly as dotted tuples. That can be implemented entirely within the toolstack, and could even allow wildcarding of tuple components to efficiently express cpumasks.
-- Keir
Quoted 8 lines
Quoted 1 line
I'd really like to see us enumerate pCPUs in a sensible order so that
it's
Quoted 2 lines
easier to see the topology. It should be nodes.sockets.cores{.threads}, leaving gaps for missing execution units due to hot plug or non power of
two
Quoted 2 lines
packing.
Right now we're inconsistent in the enumeration order depending on howthe
Quoted 8 lines
Quoted 1 line
BIOS has set things up. It would be great if someone could volunteer to
fix
Quoted 1 line
this...
Even better would be to have pCPUs addressable and listable explicitly as dotted tuples. That can be implemented entirely within the toolstack, and could even allow wildcarding of tuple components to efficiently express cpumasks.
Yes, I'd certainly like to see the toolstack support dotted tuple notation.
However, I just don't trust the toolstack to get this right unless xen has already set it up nicely for it with a sensible enumeration and defined sockets-per-node, cores-per-socket and threads-per-core parameters. Xen should provide a clean interface to the toolstack in this respect.
Ian
Quoted 11 lines · Ian Pratt
Quoted 4 lines
Even better would be to have pCPUs addressable and listable explicitly as dotted tuples. That can be implemented entirely within the toolstack, and could even allow wildcarding of tuple components to efficiently express cpumasks.
Yes, I'd certainly like to see the toolstack support dotted tuple notation.
However, I just don't trust the toolstack to get this right unless xen has already set it up nicely for it with a sensible enumeration and defined sockets-per-node, cores-per-socket and threads-per-core parameters. Xen should provide a clean interface to the toolstack in this respect.
Xen provides a topology-interrogation hypercall which should suffice for tools to build up a {node,socket,core,thread}<->cpuid mapping table.
-- Keir
Quoted 6 lines · Andre Przywara
Keir, I will check if dropping the node containment in the CPU overcommitment case is an option, but what would be the right strategy in that case?
Warn the user?
Don't contain at all?
Contain to more than onde node?
I would suggest simply don't contain at all (i.e., keep equivalent numa=off behaviour) would be safest.
-- Keir
Quoted 10 lines
Quoted 6 lines
Keir, I will check if dropping the node containment in the CPU overcommitment case is an option, but what would be the right strategy in that case?
Warn the user?
Don't contain at all?
Contain to more than onde node?I would suggest simply don't contain at all (i.e., keep equivalent numa=off
behaviour) would be safest.
I disagree. In systems with 2 nodes it will use all nodes, which is the same as your propose[*]. In systems with more nodes it will do placement to some subset. Note that systems with >2 nodes generally have stronger NUMA effects and these are exactly the systems where node placement is a good thing.
[*] note that numa=off is quite different from just disabling node placement. If node placement is disabled we still get the benefit of memory striping across nodes, which at least avoids some performance cliffs.
Ian
Quoted 2 lines
Quoted 1 line
Quoted 1 line
Andre Przywara <[email protected]> 09.11.09 16:02 >>>
BTW: Shouldn't we set finally numa=on as the default value?
I'd say no, at least until the default confinement of a guest to a single node gets fixed to properly deal with guests having more vCPU-s than a node's worth of pCPU-s (i.e. I take it for granted that the benefits of not overcommitting CPUs outweigh the drawbacks of cross-node memory accesses at the very least for CPU-bound workloads).
Jan
Quoted 8 lines
Quoted 2 lines
Quoted 1 line
Quoted 1 line
Andre Przywara <[email protected]> 09.11.09 16:02 >>>
BTW: Shouldn't we set finally numa=on as the default value?
I'd say no, at least until the default confinement of a guest to a single node gets fixed to properly deal with guests having more vCPU-s than a node's worth of pCPU-s (i.e. I take it for granted that the benefits of not overcommitting CPUs outweigh the drawbacks of cross-node memory accesses at the very least for CPU-bound workloads).
What default confinement? I thought guests had an all-pCPUs affinity mask be default?
I suspect we will get benefits enabling NUMA even if all the guests have all-pCPUs affinity masks: all guests will have memory stripped across all nodes, which is likely better than allocating from one node and then the other. Obviously assigning VMs to node(s) and allocating memory accordingly is the best plan.
Ian
Quoted 11 lines
Quoted 9 lines
Quoted 3 lines
Ian Pratt <[email protected]> 10.11.09 02:46 >>>
Quoted 1 line
Quoted 1 line
Andre Przywara <[email protected]> 09.11.09 16:02 >>>
BTW: Shouldn't we set finally numa=on as the default value?
I'd say no, at least until the default confinement of a guest to a single node gets fixed to properly deal with guests having more vCPU-s than a node's worth of pCPU-s (i.e. I take it for granted that the benefits of not overcommitting CPUs outweigh the drawbacks of cross-node memory accesses at the very least for CPU-bound workloads).
What default confinement? I thought guests had an all-pCPUs affinity mask be default?
Not with numa=on (see also Andre's post to this effect): The guest will get assigned to a node, and its affinity set to that node's CPUs.
Jan
Quoted 5 lines · Jan Beulich
Quoted 2 lines
What default confinement? I thought guests had an all-pCPUs affinity mask be default?
Not with numa=on (see also Andre's post to this effect): The guest will get assigned to a node, and its affinity set to that node's CPUs.
...And if it didn't, striping would not happen. In fact iirc the default NUMA allocation policy for an all-pcpus domain is in some respects pessimal:
vcpu0's initial node gets drained of memory first. I.e., you get *less* 'striping' than you could with numa=off where you might at least get lucky.
-- Keir
Quoted 8 lines · Jan Beulich
Quoted 2 lines
Quoted 1 line
Quoted 1 line
Andre Przywara <[email protected]> 09.11.09 16:02 >>>
BTW: Shouldn't we set finally numa=on as the default value?
I'd say no, at least until the default confinement of a guest to a single node gets fixed to properly deal with guests having more vCPU-s than a node's worth of pCPU-s (i.e. I take it for granted that the benefits of not overcommitting CPUs outweigh the drawbacks of cross-node memory accesses at the very least for CPU-bound workloads).
If this would be fixed (e.g., turn off node locality entirely by default for domains which will not fit into a single node) then I think we could consider numa=on by default.
-- Keir
PATCH 1/1 tools: avoid over-commitment if numa=on
In reply to Jan BeulichQuoted 8 lines · Jan Beulich
Quoted 2 lines
Quoted 1 line
Quoted 1 line
Andre Przywara <[email protected]> 09.11.09 16:02 >>>
BTW: Shouldn't we set finally numa=on as the default value?
I'd say no, at least until the default confinement of a guest to a single node gets fixed to properly deal with guests having more vCPU-s than a node's worth of pCPU-s (i.e. I take it for granted that the benefits of not overcommitting CPUs outweigh the drawbacks of cross-node memory accesses at the very least for CPU-bound workloads).
That sounds reasonable.
Attached a patch to lift the restriction of one node per guest if the number of VCPUs is greater than the number of cores / node. This isn't optimal (the best way would be to inform the guest about it, but this is another patchset ;-), but should solve the above concerns.
Please apply,
Andre.
Signature · 9 lines
Andre Przywara AMD-Operating System Research Center (OSRC), Dresden, Germany Tel: +49 351 448 3567 12 ----to satisfy European Law for business letters: Advanced Micro Devices GmbH Karl-Hammerschmidt-Str. 34, 85609 Dornach b. Muenchen Geschaeftsfuehrer: Andrew Bowd; Thomas M. McCoy; Giuliano Meroni Sitz: Dornach, Gemeinde Aschheim, Landkreis Muenchen Registergericht Muenchen, HRB Nr. 43632
# HG changeset patch
# User Andre Przywara <[email protected]> # Date 1259594006 -3600
# Node ID bdf4109edffbcc0cbac605a19d2fd7a7459f1117 # Parent abc6183f486e66b5721dbf0313ee0d3460613a99 allocate enough NUMA nodes for all VCPUs
If numa=on, we constrain a guest to one node to keep it's memory accesses local. This will hurt performance if the number of VCPUs is greater than the number of cores per node. We detect this case now and allocate further NUMA nodes to allow all VCPUs to run simultaneously.
tools/python/xen/xend/XendDomainInfo.py +10 -4
@@ -2637,8 +2637,7 @@ nodeload[i] = int(nodeload[i] * 16 / len(info['node_to_cpu'][i])) else: nodeload[i] = sys.maxint - index = nodeload.index( min(nodeload) ) - return index + return map(lambda x: x[0], sorted(enumerate(nodeload), key=lambda x:x[1])) info = xc.physinfo() if info['nr_nodes'] > 1:
@@ -2648,8 +2647,15 @@ for i in range(0, info['nr_nodes']): if node_memory_list[i] >= needmem and len(info['node_to_cpu'][i]) > 0: candidate_node_list.append(i) - index = find_relaxed_node(candidate_node_list) - cpumask = info['node_to_cpu'][index] + best_node = find_relaxed_node(candidate_node_list)[0] + cpumask = info['node_to_cpu'][best_node] + cores_per_node = info['nr_cpus'] / info['nr_nodes'] + nodes_required = (self.info['VCPUs_max'] + cores_per_node - 1) / cores_per_node + if nodes_required > 1: + log.debug("allocating %d NUMA nodes", nodes_required) + best_nodes = find_relaxed_node(filter(lambda x: x != best_node, range(0,info['nr_nodes']))) + for i in best_nodes[:nodes_required - 1]: + cpumask = cpumask + info['node_to_cpu'][i] for v in range(0, self.info['VCPUs_max']): xc.vcpu_setaffinity(self.domid, v, cpumask) return index
_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel